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This  technical  report  represents  the  test  plan  for  conducting  the 
Concept  Validation,  Combined  Development  Test  and  Evaluat ion /Operational 
Evaluation  of  the  Compass  Preview  Test  Bed  System.  This  test  plan 
will  be  implemented  by  test  procedures  developed  under  a  follow-on 
contract  F30602-80-C-0210,  Compass  Preview  Test  Support  whose  purpose  is 
to  conduct  the  test  as  per  this  test  plan  and  write  a  final  test  report. 
This  test  plan  has  been  reviewed  and  endorsed  by  all  the  members  of  the 
Test  Planning  Working  Group  (TPWG)  who  provided  technical  inputs  to  the 
author . 

This  effort  is  relevant  to  the  RADC  Technology  Plan  "Recce/Intell" 
since  it  provides  a  test  plan  for  testing  a  digital  image  exploitation 
system. 
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SECTION  I.  TEST  BACKGROUND  AND  PURPOSE 

1.  Background  and  Authority. 

Increasingly  large  volumes  of  digital  imagery  have  been  and  are  continuing 
to  be  collected  in  support  of  requirements  levied  by  the  U.S.  intelligence 
community.  Unfortunately,  the  large  amounts  of  imagery  have  consistently 
overloaded  the  exploitation  capabilities  of  organizations  designed  to  sat¬ 
isfy  these  requirements.  At  the  SAC/544th  SIW  exploitation  facility  imple¬ 
mentation  of  the  Program  Assisted  Console  Evaluation  and  Review  (PACER) 
System  has  helped  alleviate  the  problems  of  intelligence  data  handling. 
However,  the  current  imagery  exploitation  devices  are  limited  to  exploita¬ 
tion  of  hard  copy  photographic  film,  and  have  no  capability  for  handling 
or  exploiting  electronic  imagery  in  a  time  responsive  manner.  The  COMPASS 
PREVIEW  Program  is  designed  to  provide  a  near-real-time  digital  image 
exploitation  capability  which  will  improve  the  quality  and  quantity 
(throughput)  of  the  resulting  intelligence.  COMPASS  PREVIEW  will  test  an 
exploitation  capability  comparable  in  technology  and  efficiency  to  that 
found  in  current  collection  and  data  handling  systems. 

The  need  for  an  advanced  exploitation  capability  was  recognized  in  the 
late  1960fs  and  culminated  in  SAC  ROC  22-69,  dtd  28  Oct  1969,  and  entitled 
,f Advanced  High  Resolution  Photo  Interpretation  Device.11  This  was  subse¬ 
quently  supplemented  by  a  SAC  Operational  Concept  dtd  15  Jan  71  entitled 
COMPASS  PREVIEW  and  a  SAC  Statement  of  requirements  for  COMPASS  PREVIEW  in 
April  of  1975.  The  original  ROC  has  been  updated  in  draft  form  in  March 
of  1978. 

Overall  program  direction  is  provided  in  PMD  R-P5021 (7)64750F,  dtd  30  May 
1980.  This  PMD  directs,  "Concept  Validation  testing  will  be  conducted  as 
a  Combined  DT&E/ Operational  Evaluation  IAW  the  procedures  of  AFR  80-14." 
Following  the  completion  of  testing  the  system  will  be  removed  from  SAC 
by  the  implementing  command. 

2.  Test  Purpose. 

In  accordance  with  PMD  direction,  CPTB  testing  will  be  conducted  using 
AFR  80-14  procedures.  The  DT&E  portion  will  be  conducted  to  demonstrate 
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that  the  engineering  design  and  development  are  complete,  that  design 
risks  have  been  minimized  and  that  the  system  will  meet  engineering  and 
operational  specifications*  The  operational  evaluation  will  estimate 
the  system  operational  effectiveness  and  suitability,  and  identify  any 
operational  deficiencies  and  need  for  modifications.  The  concept  valida¬ 
tion  will  provide  data  and  an  experiential  basis  for  evaluation  of  future 
USAF  soft  copy  exploitation  requirements. 

This  plan  defines  a  test  structured  to  accomplish  these  purposes  within 
the  guidelines  of  the  regulation,  taking  cognizance  of  the  fact  that 
"operational  testing  should  be  separate  from  developmental  testing"  and 
that  combining  is  acceptable  only  to  reduce  delay  and  to  economize  the 
use  of  resources.  The  combination  here  is  limited  to  that  portion  of 
DT&E  which  involves  operational  (as  opposed  to  engineering)  specifications 
and  which  can  benefit  from  the  operational  test  requirements  to  "assess 
system  performance  where  the  complete  system  is  tested  and  evaluated 
against  operational  criteria  (requirement  and  employment  concepts)  by 
personnel  with  the  same  qualifications  as  those  who  will  operate  . . .  the 
system  when  deployed."  The  test  structure  is  also  designed  to  facilitate 
participation  by  a  number  of  USAF  commands  in  addition  to  SAC  and  to  pro¬ 
vide  data  regarding  soft  copy  exploitation  applicability  to  tactical  as 
well  as  strategic  mission  requirements. 


3.  Program  Schedule  and  Milestones. 

There  are  two  major  program  milestones  which  directly  influence  the  test 
program  milestones.  These  are: 

(1)  Delivery  of  CPTB  to  SAC  1  JUN  1980 

(2)  Completion  of  Final  Acceptance  Test  1  OCT  1980 


The  major  milestones  of  the  test  program  itself  are: 


Final  Approved  Test  Plan 

Test  Procedures  Complete  Draft 

Test  Force  Training 

Start  Pretest  Preparation  Period 

Dry  Run  Test 

Part  1  Test 


30  MAY  1980 
1  OCT  1980 
6-17  OCT  1980 
1  OCT  1980 
19  JAN  1980 
9  FEB  1981 


3  - 


Part  2  Test 

Interim  Report,  Part  1 
Part  3  Test 

Interim  Report,  Part  2 
END  Test 

Interim  Report,  Part  3 
Final  Test  Report 


30 

MAR 

1981 

1 

MAY 

1981 

1 

JUN 

1981 

29 

JUN 

1981 

30 

OCT 

1981 

11 

DEC 

1981 

28 

FEB 

1982 

Figure  1  provides  a  graphic  display  of  the  overall  schedule  showing  addi¬ 
tional  detail  in  certain  portions  of  the  program  (i.e. ,  image  selection, 
procedures  drafts).  Section  A-4,  Annex  A  provides  detailed  schedule  for 
participation  by  all  commands  in  each  test  part. 


4.  Test  Item  Description. 

The  CPTB  is  designed  to  provide  the  imagery  interpreter/analyst  with  an 
electronic  imagery  exploitation  capability,  comparable  in  technology  and 
efficiency  to  the  collection  and  data  handling  system  now  supporting 
imagery  based  reconnaissance.  It  is  designed  to  enable  the  exploitation 
process  to  provide  more  complete  and  accurate  intelligence  products  in  a 
time- responsive  manner.  The  CPTB  design  provides  signif icant ly  enhanced 
capabilities  by  integrating  viewing  and  analytical  functions  within  a 
single  exploitation  device.  CPTB  provides  capabilities  for  measurement, 
geometric  correct  ion/rectification  and  stereo  pair  matching;  image  enhance¬ 
ment,  and;  interactive  image  graphics.  It  is  responsive  to  direct  inter¬ 
preter  control  and  by  integration  with  an  on-line  data  base,  enables 
rapid  intelligence  production  in  time  dominated/crisis  situations. 


The  major  hardware  components  of  the  CPTB  system  are  (1)  the  digital  image 
viewer  unit,  (2)  the  refresh  memory  units,  (3)  magnetic  tape  units, 

(4)  system  and  station  processors,  (5)  image  interface  buffer,  and 
(6)  adjacent  consultation  station. 


The  digital  image  viewer  unit  is  the  portion  of  the  CPTB  seen  by  the 
interpreter /analyst  and,  as  the  working  console,  is  the  tangible  element. 
Figure  2  shows  the  DIVU  and  its  major  components.  Figure  3  shows  the 
physical  configuration  of  the  system  as  installed  at  SAC. 


ioasn«iA*«j  £0!  DidY 


Figure  2.  DIGITAL  IMAGE  VIEW  UNIT 


DIGITAL  IMAGE 
VIEWER  ASSEMBLY 
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There  are  75  CPTB  functions.  These  75  fall  into  two  groups  comprising  nine 
major  categories,  as  follows: 

IMAGE  FUNCTIONS 

(1)  Image  Selection  Category  —  those  functions  which  enable  the  image 

interpreter  (II)  to  identify  and  select  the  desired  frames/ installation 


to  be  displayed  including: 
New  Frame  Queue 
Saved  Frame  Queue 
Alternate  Frame 
Next  Frame 


Save  Frame 

Restore  Frame 

Release  Frame 

Next  Installation 

Current  Frame  Installation  List 


(2)  Image  Mode  Category  —  functions  which  permit  the  selected  image  to  be 
displayed  and  viewed  in  the  most  advantageous  manner  including: 


Full  Frame 
Mono  Mode 
Stereo  Mode 
Stereo  exchange 
Stereo  Invert 
Rotate  Image 


Magnification  Vernier 
Image  Translation 
Optimum  Presentation 
Left  Image  Select 
Right  Image  Select 


(3)  Subpictures  Category  —  functions  which  enable  the  II  to  quickly  and 
easily  focus  on  a  small  area  for  more  detailed  analysis  including: 

Build  Subpicture  Delete  Subpicture  Set 

Display  Subpicture  Subpicture  Look  Ahead  Mode 

Next  Entry-Subpics  Overview  Reference 

Prior  Entry-Subpics  Save 

Delete  Entry-Subpics  Restore  Subpicture 

Step  Resolution 

(4)  Image  Manipulation  Category  —  functions  which  enable  the  II  to  correct 
anomalies  and  to  enhance  the  image  to  improve  the  extraction  of  detailed 
information  including: 

Gray  Level  Control  Geometric  Correction 

Image  Enhance  Geometric  Alignment 


_ —  j  ,  ^ 


(5)  Image  Computation  Category  —  functions  which  enable  the  II  to  make 
measurements  and  correct  location  errors  including: 

Compute  Coordinate  Image  Measurements 

Position  Cursor  at  Coordinate  Coverage  Correction 


(6)  Image  Graphics  Category  —  functions  which  enable  the  II  to  tally 
counts  and  to  annotate  images  including: 

Count  Annotations  Display  All  Graphics 

Display/Installation  Symbols  Display/Fixed  Annotations 

Line  Annotations  Image  Hard  Copy 

Delete  Annotations  Target  Materials  Grid 


Delete  Annotations 
Text  Annotations 


DATABASE  FUNCTIONS 

(7)  Provide  Information  Category  —  functions  providing  data  that  facilitate 
interpretation  and  reporting: 


Current  Frame  Installation  List 
Next  Installation 
Target  Specs 
EEI  Requirements 
Latest  PI  Reports 


Latest  OB 
PI  Des/PI  Notes 
Collateral  Intelligence 
AI  Reports 
Quick  Look 


(8)  Create  Information  Category  —  functions  which  enable  the  II  to  update 
information  in  the  PACER  data  base  including: 

MIPIR  Coverage  Not  Available 


MIPIR 

Report  Create 
New  Installation 
Cloud  Cover 

Rest  of  Frame  Cloud  Covered 
ID  Only 
Index  Only 


No  Apparent  Change 
Update  PI  Notes 
Photo  Description  Update 
Flash  Report 

Installation  description 
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(9)  Support  Functions  Category  —  functions  which  enable  the  II  to  com¬ 
municate  directly  with  PACER  to  which  provide  miscellaneous  capabil¬ 
ities  in  support  of  exploitation  requirements. 

Data  Base  Queries  Station  Status  Line 

Query  Summary /Results  Miscellaneous 

Query  —  Next  Entry  Print  Display 

Scope  and  Limiting  Factors. 

As  indicated  by  the  complex,  compound  title  of  this  test,  it  is  an  inquiry 
of  much  broader  than  usual  scope,  incorporating  the  usual  concerns  of  the 
developing  command  (RADC)  and  the  primary  using  command  (SAC)  with  addi¬ 
tional  requirements  and  applications  of  the  other  participating  commands 
(FTD,  PACAF,  TAC,  and  USAFE)  and  the  generalized  examination  of  efficacy 
of  soft  copy  exploitation  approaches  embodied  in  the  system.  Simultane¬ 
ously,  however,  the  scope  of  possible  testing  is  severely  restricted  by 
a  number  of  temporal,  physical,  and  technical  constraints. 

The  temporal  constraints  are  a  joint  result  of  a  limitation  of  the  total 
available  time  for  test  preparation  (of  soft  copy  materials)  and  test 
conduct  and  the  fact  that  there  is  only  one  CP  work  station  thus  neces¬ 
sitating  that  all  work  be  accomplished  in  series  rather  than  the  more 
time  conserving  parallel  mode. 

The  physical  constraints  are  focused  around  the  necessity  to  prepare  for 
and  conduct  the  test  in  the  midst  of  an  operational  environment.  The  host 
command  (SAC/544th  SIW)  must  maintain  normal  operations  and  this  places 
severe  limitations  on  space,  and  schedule. 

Other  aspects  of  the  system  which  limit  the  test  are:  (a)  the  main  image 
display  cannot  be  seen  simultaneously  by  test  interpreter  and  test  monitor 
making  precision  data  collection  difficult,  (b)  Training/practice  time  is 
limited  by  single  work  station  making  it  difficult  to  achieve  desirable 
proficiency  levels  for  valid  comparison  to  hard  copy  methods,  and  (c)  System 
capabilities  are  as  designed  for  a  specific  operational  requirement 
(i.e.  SAC)  and  are  not  general  and  flexible  enough  to  facilitate  detailed 
examination  of  broader  soft  copy  exploitation  concepts. 


6.  Test  Location. 

The  test  will  be  conducted  at  SAC  Headquarters,  Offutt  AFB,  NE  in  secure 
spaces  provided  by  the  544th  SIW. 

7.  Test  Management. 

The  test  is  controlled  by  a  Test  Planning  Working  Group  (TPWG)  consisting 
of  representatives  of  the  developing  command,  using  commands,  system  con¬ 
tractor,  and  test  contractor.  Table  1  lists  the  members  of  the  TPWG. 

The  chairman  of  the  TPWG  is  the  RADC  Test  Director,  Mr.  Peter  Costianes. 
Day-to-day  operation  will  be  under  the  control  of  the  544th  SIW  Test 
Director  and  the  test  contractor  on-site  Test  Manager. 

The  Test  Team  is  a  task  organized  group  which  has  been  structured  to  most 
effectively  accomplish  the  preparation  for,  conduct  of,  and  reporting  of 
the  test,  without  regard  for  command  structure  or  contractual  relation¬ 
ships.  A  review  of  Section  II  PAR  7  reveals  the  complexity  of  the  over¬ 
all  test  organization  and  its  associated  lines  of  authority/responsibility. 
It  is  the  purpose  of  this  paragraph  and  Figure  4  to  delineate  the  structure 
of  the  working  relationships  necessary  to  efficiently  accomplish  the  test 
activity.  It  places  the  544th  SIW  Test  Director  in  the  key  role  of  coor¬ 
dinating  and  managing  the  day-to-day  activity.  The  contractor  on-site 
test  manager  will  act  as  his  deputy,  responsible  for  direct  management  of 
test  preparation  and  conduct  activity.  Overall  test  program  control  is 
the  responsibility  of  the  RADC  test  director.  The  participating  command 
representatives  will  constitute  an  advisory  panel  in  support  of  the  RADC 
test  director  and  will  be  responsible  for  insuring  the  applicability  of 
test  conditions  to  their  commands 1  operational  environments  and  for  the 
interpretation  and  extrapolation  of  test  results  to  their  commands* 
operational  requirements.  Other  on-site  members  of  the  Test  Team  will  be 
the  ATC  representative  (responsible  for  TII  training),  the  Test  Material/ 
Data  Control  Librarian,  Hard  copy  monitor.  Soft  copy  monitor,  and  SAC/INCRP. 

8.  Training  Requirements. 

The  training  of  test  team  members  falls  into  three  categories;  training  of 
test  staff,  training  of  system  operators  and  training  of  test  interpreters. 
The  test  staff  will  receive  two  weeks  of  system  training  during  the  month 
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Figure  4,  Test  Team  Functional  Structure  Diagram 
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of  September  1980  conducted  by  the  system  contractor  as  required  by  the 
prime  contract.  System  operators  will  also  be  trained  by  the  system 
contractor.  The  basis  for  this  training  will  be  in  the  form  of  on-the- 
job  training  working  with  system  contractor  maintenance  personnel.  Test 
interpreters  will  receive  special  training,  prepared  and  conducted  by 
the  test  staff  personnel,  immediately  prior  to  participation  in  each  part 
of  the  test  program.  Training  for  Part  1  will  be  one  week;  training  for 
Part  2  will  be  two  weeks,  training  for  Part  3  will  be  two  weeks. 

9.  Environmental  Impact. 

There  is  no  known  or  anticipated  environmental  impact  resulting  from  the 
conduct  of  this  test. 

10.  Safety. 

There  are  no  known  or  anticipated  special  safety  requirements  involved  in 
the  conduct  of  this  test.  Prior  to  testing,  a  safety  survey  will  be  con¬ 
ducted  by  a  qualified  safety  engineer.  If  any  potential  hazards  are 
identified  during  the  pre-test  survey  or  during  the  test  phase,  they  will 
be  documented  and  evaluated  prior  to  further  testing.  A  copy  of  the 
documentation  and  action  will  be  retained  in  the  project  file. 

11.  Security. 

Participation  in  this  test  will  require  access  to  special  compartmented 
information  (SCI).  Test  material  preparation,  test  conduct  and  initial 
data  reduction  will  be  confined  to  the  system  high  facilities  of  the  544th 
SIW.  Data  analysis  and  reporting  will  be  handled  at  an  unclassified  level 
with  special  classified  annexes  to  the  test  report,  if  required.  Access 
to  the  test  facility  for  test  participants  and  all  visit  requests  will  be 
handled  through  the  SAC  point  of  contract;  SAC/INCRP,  Vice  Chairman, 
COMPASS  PREVIEW  TPWG. 

12.  Release  of  Information. 

Release  of  any  information  regarding  the  system,  the  test,  or  the  test  re¬ 
sults  must  be  approved  by  RADC/IRRS.  The  RADC  Test  Director  has  releasing 
authority  for  all  unclassified  information  and  all  test  information  re¬ 
leases  will  be  cleared  through  him. 
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13. 


Intelligence. 

There  is  no  known  or  anticipated  intelligence  threat  created  by  the 
conduct  of  this  test. 


j 
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SECTION  II.  METHOD  OF  ACCOMPLISHMENT 
!•  System  Operational  Concept  and  Maintenance  Concept. 

A.  Operational  Concept.  The  SAC  operational  concept  for  COMPASS  PREVIEW 
was  written  in  1971.  Changes  in  program  direction  have  expanded  the 
area  of  concern  beyond  that  of  SAC  alone.  No  single  clear  statement 
of  an  operational  concept  is  applicable  for  the  purpose  of  this  test. 

One  of  the  purposes  of  the  system  development  and  of  the  test  program 
is,  as  stated  in  the  system  specification,  (PAR  3.1.3  Missions)  "The 
mission  of  the  CPPS  (now  CPTB)  is  to  provide  an  advanced  digital  ima¬ 
gery  exploitation  capability  for  evaluation  ...  to  develop  future 
concept  of  operations  for  exploiting  digital  imagery. "  It  has  been 
further  determined  by  the  TPWG  that  the  test  program  must  provide  a 
basis  in  the  form  of  both  performance  data  and  image  interpreter 
experience  upon  which  the  participating  commands  can  draw  to  formulate 
operational  concepts  appropriate  to  their  specific  mission  requirements. 

B.  Maintenance  Concept.  Because  of  the  nature  and  purpose  of  the  CPTB 
development  there  is  no  formal  documentation  of  a  maintenance  concept. 
The  system  will  be  maintained  under  a  special  support  contract  with 
the  system  developer  to  provide  two  maintenance  personnel  on-site  at 
SAC  for  the  duration  of  the  test. 

2.  Critical  Questions. 

There  are  a  number  of  critical  issues  and  areas  of  risk  which  underlie  the 
test  objectives.  These  stem  from  the  diversity  of  requirements  represented 
by  the  various  potential  users  and  the  current  state  of  the  technologies 
involved  in  the  soft  copy  imagery  exploitation.  Collectively,  they  shape 
both  the  objectives  themselves  and  the  test  approach  selected  to  address 
them. 

Implicit  in  each  issue  is  the  comparison  with  or  improvement  over  current 
methods  employed  to  accomplish  image  exploitation  tasks.  There  are  three 
major  categories  of  concern:  (1)  viewing  digital  imagery,  (2)  data  base 
integration,  and  (3)  the  system  as  a  whole. 
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A.  Viewing  Digital  Imagery. 

(1)  Does  the  image  interpreter  accept  the  electronic  imagery  display 
as  a  comfortable  mode  of  viewing  imagery? 

(2)  Is  the  image  interpreter  satisfied  with  his  ability  to  create  a 
stereo  presentation  of  the  imagery  using  the  system  functions 
available  to  him? 

(3)  Do  the  subpicture  modes  of  viewing  satisfy  the  requirement  to 
view  imagery  at  different  magnifications? 

(4)  Does  the  image  interpreter  understand  the  capabilities  for  image 
enhancement  and  how  to  use  them  to  enhance  his  productivity? 

(3)  Does  the  highly  integrated  (imagery,  graphics,  alphanumeric)  and 
interactive  (sonic  pen,  joy  stick,  etc.)  display  mode  overload 
or  assist  the  one-man  exploitation  concept? 

B.  Integrated  Data  Base. 

(1)  How  effective  is  the  integration  of  image  functions  and  intelli¬ 
gence  handling/display  functions  in  the  satisfaction  of  image 
exploitation  requirements? 

(2)  How  effective  are  the  integrated  features  such  as  the  input  sort¬ 
ing/queuing  or  imagery  and  installation  information  in  increasing 
throughput? 

C.  System. 

(1)  Is  the  CP  system  reliable,  maintainable,  and  logistically 
supportable? 

(2)  What  capabilities  should  be  added  to  or  deleted  from  the  CP 
approach  to  soft  copy  exploitation  to  enhance  its  operational 
effectiveness/ suitability . 

3.  Test  Methodology. 

The  actual  testing  is  divided  into  three  parts:  Part  1,  an  evaluation  of 
specific  soft  copy  exploitation  capabilities;  Part  2,  a  controlled  compari¬ 
son  of  hard  copy  vs.  soft  copy  exploitation  methods  as  applied  to  DIAM  57-7 
reporting  tasks;  Part  3,  an  examination  of  soft  copy  exploitation  concepts 
in  an  operational  environment. 


Part  1  will  consist  of  four  subtests:  (1)  Stereo,  (2)  Subpictures, 

(3)  Measurements,  and  (4)  Enhancement.  Together,  they  are  designed  to 
provide  detailed  data  regarding  the  effectiveness,  utility,  and  opera¬ 
bility  of  specific  CPTB  functions.  The  first  two  subtests  (Stereo, 
Subpictures)  will  utilize  a  direct  hard  copy  comparison  condition. 

Measurements  will  employ  a  ground  truth  standard  but  no  hard  copy  com¬ 
parison.  Enhancements  will  involve  a  structured  demonstration  contrast¬ 
ing  enhanced  and  un-enhanced  soft  copy  imagery.  The  four  subtests  will 
be  conducted  as  a  “block"  with  tasks  from  all  tests  interspersed  and 
scheduled  so  that  each  pair  of  test  interpreters  will  complete  their  par¬ 
ticipation  in  two  weeks  (1  week  training,  1  week  test).  In  general,  the 
response  measure  employed  will  involve  the  speed  and  correctness  of  per¬ 
formance  of  specific  image  interpretation  subtasks.  These  subtasks  will 
range  from  such  specific  elements  as  "time  to  achieve  stereo  display"  to 
more  generalized  activities  such  as  "target  detection  time",  "accuracy  of 
identification",  etc.  By  collecting  data  at  this  level  of  detail,  the 
results  will  be  interpretable  in  the  context  of  other  user  appl ications 
using  the  same  "building  block"  subtasks  to  accomplish  different  missions. 

The  general  test  format  is  similar  to  a  controlled  experiment  in  which 
each  image  interpreter  will  perform  the  functions  repeatedly,  under  con¬ 
trolled  conditions,  over  a  representative  set  of  images.  Each  image  in¬ 
terpreter  will  see  all  images  (half  on  CPTB,  half  in  hard  copy,  stereo 
and  subpictures)  and  the  results  will  be  pooled  to  provide  a  reliable, 
stable  estimate  of  the  level  of  performance. 

The  major  purpose  of  the  stereo  test  is  to  investigate  the  acceptability, 
ease  of  use  and  functional  utility  of  the  CPTB  capability  for  unaided 
viewing  as  compared  to  the  conventional  stereoscope.  Imbedded  in  this 
process  is  an  assessment  of  the  image  manipulation  functions  which  can 
be  used  to  improve  the  stereo  view  and  generally  aid  the  interpreter  in 
the  process  of  obtaining  the  best  presentation.  The  test  will  require  a 
specially  selected  set  of  stereo  pairs  which  are  generally  typical  of  the 
range  of  imagery  collected  and  which  contain  a  widely  distributed  set  of 
objects  whose  relative  height  can  be  reliably  determined  only  in  a  stereo 
presentation.  The  test  schedule  will  control  image  presentation  order  of 
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primary  frames.  The  first  task  will  be  to  achieve  stereo  display.  Once 
this  is  accomplished  and  the  associated  data  recorded,  further  measures  of 
accuracy  of  interpretation  and  application  of  other  manipulations  will  be 
obtained . 

The  major  purpose  of  the  subpictures  test  is  to  investigate  the  contribu¬ 
tion  to  the  interpretation  process  made  by  the  CPTB  capabilities  to  alter 
the  size/resolution  of  the  image  presented  to  the  interpreter.  At  issue 
here  are  both  the  initial  stage  of  the  exploitation  process,  finding  the 
target  on  the  frame,  including  search  tasks,  and  final  stages  of  detailed 
analysis,  where  magnification  is  required  to  discern  fine  details  necessary 
for  a  complete  interpretation.  The  test  will  require  a  selected  set  of 
frames  which  are  representative  of  the  range  of  target  size,  density,  and 
complexity  typically  encountered.  Initial  presentation  in  the  CPTB  condi¬ 
tion  will  be  alternately  full  frame  or  optimum  presentation  as  governed  by 
a  random  schedule.  Presentation  in  the  hard  copy  condition  will  be  in  the 
standard  SAC  format.  After  initial  measures  of  target  location  time/ 
accuracy,  a  second  random  schedule  of  subpicture  options  will  be  employed 
and  measures  of  speed  and  accuracy  of  detailed  analysis  obtained. 

The  major  purpose  of  the  measurement  test  is  to  assess  the  usability  and 
applied  accuracy  of  the  measurement/computation  functional  capabilities  of 
the  CPTB.  The  question  here  is  not  one  of  absolute  accuracy  or  the  correct¬ 
ness  of  the  various  algorithms  supplied  in  the  system  software  (since  that 
has  been  established  in  earlier  testing)  but  rather  to  examine  the  accuracy 
attainable  in  practice;  accuracy  of  measurements  made  by  an  image  interpre¬ 
ter  on  good  quality  imagery  taken  over  controlled  ranges.  The  test  exer¬ 
cises  will  require  that  certain  measures  be  made  in  various  areas  of  the 
frame  and  under  different  conditions  of  image  manipulation  including 
coverage  correction,  rotated/translated  views,  and  geometrically  corrected 
frames.  Also  included  in  this  test  will  be  exercises  involving  coordinate 
determination  of  new  "targets’*  and  the  verification  of  known  target  coor¬ 
dinates  . 

The  major  purpose  of  the  enhancement  test  is  to  demonstrate  the  contribu¬ 
tion  to  image  interpretability/exploitability  made  by  the  CPTB  functional 
capabilities  for  image  enhancement.  Specifically,  this  test  will  seek  to 
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demonstrate  both,  how  much  image  enhancement  helps  the  interpreter  and 
how  best  to  use  these  capabilities.  This  test  will  be  conducted  as  a 
mixture  of  controlled  exercises  where  the  interpreter  will  use  a  speci¬ 
fied  degree  and  type  of  enhancement,  first  viewing  the  frame  un-enhanced, 
then  applying  the  enhancement.  Each  exercise  will  involve  the  presenta¬ 
tion  of  an  installation  and  the  posing  of  questions  directed  at  "fine 
grain'1  details  of  target  features;  details  which  could  be  anticipated  to 
be  more  accurately  and  completely  reported  from  an  enhanced  image.  Data 
will  be  obtained  on  both  the  results  of  the  enhancement  and  the  interpre¬ 
ters  evaluation  of  its  effectiveness  in  facilitating  detailed  exploitation. 

Part  2,  the  DIAM  57-7  reporting  test,  will  exercise  all  the  capabilities 
of  the  CPTB  in  a  controlled  fashion  employing  typical  operational  imagery 
and  reporting  requirements.  It  is  to  be  conducted  under  controlled  condi¬ 
tions  with  some  rather  detailed  data  collection  but  the  test  exercise  is 
based  upon  an  operational  image  interpretation  task  structure  as  currently 
implemented  in  the  544th  SIW.  Furthermore,  it  is  structured  to  attend 
directly  to  the  question  of  overall  performance  enhancement  attainable 
by  the  use  of  che  CPTB.  It  is  a  direct  hard  copy /soft  copy  comparison 
test.  The  measure  to  be  employed  will  attend  to  the  quality,  timeliness, 
and  quantity  of  intelligence  production  attainable  by  the  application  of 
the  CPTB  as  compared  to  the  current  SAC  methods. 

The  specific  approach  to  this  test  will  require  the  careful  selection  of 
imagery.  Test  control  and  the  validity /generalizability  of  the  results 
depend  upon  the  representative  nature  of  the  image  sample.  The  test  will 
be  conducted  using  preselected  imagery  and  an  edited  version  of  the  PACER 
data  base,  sepcially  constructed  to  be  time  correlated  with  the  selected 
imagery.  The  interpreter's  task  will  involve  all  of  the  major  steps  of 
the  normal  operating  procedure  including  frame  selection,  installation 
location,  preparation  for  exploitation,  report  preparation,  data  base  up¬ 
date,  and  preparation  of  graphics.  The  complete  sequence  will  be  accom¬ 
plished  in  both  hard  copy  and  soft  copy  modes. 

Part  3  of  the  test  has  two  major  purposes.  The  first  is  to  provide  a 
direct  comparison  of  product  quality  (or  acceptability),  timeliness,  and 
productivity  using  CPTB  as  compared  to  current  operating  procedures,  where 
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both  test  conditions  use  operational  imagery  and  the  actual  PACER  data 
base-  The  second  is  to  provide  an  evaluation  of  the  efficacy  of  the  data 
base  integration  features  of  the  CPTB  soft  copy  exploitation  concept. 

To  accomplish  this  purpose  three  conditions  must  be  evaluated:  (1)  CPTB 
as  designed,  (2)  CPTB  as  a  "soft  copy  light  table"  in  conjunction  with  a 
regular  PACER  terminal,  and  (3)  the  hard  copy  method  as  currently  employed 
in  the  544th  SIW. 


This  will  be  a  truly  operational  test  in  many  ways.  The  imagery  will  not 
be  pre-selected.  The  CPTB  will  be  employed  as  just  another  work  station 
exploiting  frames  of  imagery  in  normal  reporting  precedence  order  within 
the  area  of  functional  specialization  of  the  image  interpreter  assigned 
for  that  shift.  The  CPTB  will  be  on  line  with  PACER.  There  will  be  an 
observer/monitor  at  the  CPTB  station  but  data  collection  will  be  on  a 
strictly  "not- to- interfere"  basis.  The  comparitive  basis  for  the  test 
will  be  obtained  from  the  normal  hard  copy  exploitation  of  the  frames  of 
imagery  which  are  worked  on  the  CPTB.  The  development  of  the  requisite 
hard  copy  base  line  will  be  accomplished  as  follows:  All  imagery  will  be 
exploited  in  the  hard  copy  mode  in  the  course  of  normal  operations. 

There  will  be  no  special  assignment  or  interpreter  designation  except  for 
the  restriction  that  none  will  work  a  frame  previously  worked  on  the  CPTB. 
In  each  case  (CPTB  and  H/C)  the  PACER  update  material  resulting  from  the 
exploitation  will  be  brought  to  the  point  of  actual  entry  but  only  the 
H/C  based  version  will  actually  be  entered.  Both  versions  will  be  pro¬ 
vided  to  judges  for  evaluation  and  comparison  without  knowledge  of  which 
material  was  CPTB  generated. 

4.  Specific  Objectives. 

The  test  objectives  stem  from  the  critical  questions  and  from  the  specific 
concerns  of  the  potential  using  commands  represented  in  the  TPWG.  In  each 
case,  the  objective  has  been  stated  as  concisely  as  possible,  leaving  the 
statement  of  standards  of  performance  and  measures  of  effectiveness  to  the 
discriptions  of  the  specific  tests  designed  to  satisfy  each.  (ANNEX  A). 
The  objectives  are  grouped  under  the  heading  of  DT&E  and  operational 
evaluation.  In  addition,  each  objective  has  appended  to  it  a  reference 
to  the  specific  test(s)  from  which  data  addressing  it  will  be  obtained. 


DT&E  Objectives. 

Objectives  in  this  group  deal  with  specific  attributes/capabilities  of 
the  CPTB  approach  to  soft  copy  exploitation  and  are  addressed  mainly  in 
terms  of  data  obtained  on  the  specific  subtasks,  tasks,  or  activities 
comprising  the  job  of  an  image  interpreter. 

(1)  Evaluate  the  effectiveness  of  the  CPTB  functional  capabilities  pro¬ 
vided  in  the  Imagery  Selection  Category  which  are  designed  to  en¬ 
able  USAF  image  interpreters  to  select  and  release  specific 
imagery  frames/ installations  to  be  displayed  in  accordance  with 
established  reporting  priorities.  (Part  1  Subpictures,  Part  2). 

(2)  Evaluate  the  effectiveness  of  the  CPTB  functional  capabilities  pro¬ 
vided  in  the  Image  Mode  and  Image  Selection  Categories  which  are 
designed  to  enable  USAF  image  interpreters  to  locate  and  prepare  fo: 
exploitation  a  specific  installation  on  an  already  displayed  frame 
in  accordance  with  established  reporting  priorities.  (Part  1  Sub¬ 
pictures,  Part  2). 

(3)  Evaluate  the  effectiveness  of  the  CPTB  functional  capabilities  pro¬ 
vided  in  the  Subpictures  Category  which  are  designed  to  enable  USAF 
image  interpreters  to  more  rapidly  search  frames  of  imagery,  detect¬ 
ing  items  of  potential  intelligence  value  and  performing  detailed 
analysis.  (Part  1  Subpictures,  Part  2). 

(3a)  Evaluate  the  relative  effectiveness  of  full  frame  and  optimum  pre¬ 
sentation  modes  of  initial  imagery  display.  (Part  1  Subpictures, 
Part  2). 

(4)  Evaluate  the  effectiveness  of  the  CPTB  functional  capabilities  pro¬ 
vided  in  the  Image  Mode  Category  which  are  designed  to  enable  the 
USAF  image  interpreters  to  accomplish  exploitation  tasks  using  both 
stereo  and  mono  presentations.  (Part  1  Stereo,  Subpictures, 
Measurements,  Part  2). 

(4a)  Evaluate  the  achievement  of  stereo  model  and  the  requirements  for 
adjustments  across  the  full  field  of  view  (Part  1  Stereo). 
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(4b)  Assess  interpreter  acceptance  of  the  CPTB  capability  for  unaided 
stereo  viewing.  (Part  1  Stereo*  Part  2). 

(5)  Appraise  the  effectiveness  of  the  CPTB  functional  capabilities  pro¬ 
vided  in  the  Image  Manipulation  Category  which  are  designed  to  enable 
USAF  image  interpreters  to  increase  the  comprehensiveness  and  confi¬ 
dence  rating  of  detailed  analysis.  (Part  1  Enhancement). 

(6)  Evaluate  the  effectiveness  of  the  CPTB  functional  capabilities  pro¬ 
vided  in  the  Image  Computation  Category  which  are  designed  to  enable 
USAF  image  interpreters  to  locate  known  installations  and  determine 
the  coordinates  of  other  image  features  of  potential  intelligence 
value  and  to  determine  distances  and  dimensions  of  interest.  (Part  1 
Measurements,  Part  2). 

(6a)  Evaluate  the  effect  on  measurement/ location  accuracy  of  the  utiliza¬ 
tion  of  CPTB  functional  capabilities  in  the  Image  Computation  Cate¬ 
gory  on  geometrically  corrected  and  uncorrected  images.  (Part  1 
Measurements) . 

(6b)  Appraise  the  ease  and  accuracy  of  cursor  positioning  on  the  image 
display  using  sonic  pen,  joy  stick,  and  coordinate  input.  (Part  1 
Measurements,  Subpictures,  Part  2). 

(7)  Evaluate  the  effectiveness  of  the  CPTB  functional  capabilities  pro¬ 
vided  in  the  Image  Graphics  Category  which  are  designed  to  enable 
USAF  image  interpreters  to  annotate  imagery  through  the  application 
of  graphics,  textual  data,  count  objects  and  generate  annotated 
hard  copy  materials.  (Part  2). 

(8)  Evaluate  the  effectiveness  and  ease  of  use  of  the  CPTB  capabilities 
of  Polaroid  and  roll  film  copy  cameras  which  are  designed  to  produce 
annotated  hard  copy  materials  (graphics  and  alphanumerics)  from  the 
CPTB  displayed  imagery,  including  imagery  which  is  enlarged,  enhanced 
or  corrected.  (Part  2). 


(9)  Evaluate  the  effectiveness  of  the  CPTB  functional  capabilities  pro¬ 
vided  in  the  In format  ion /Support  Categories  which  are  designed  to 
enable  USAF  image  interpreters  to  interact  with  the  PACER  data  base 
to  obtain  textual  information  required  to  support  exploitation  tasks 
and  to  enter  new  imagery-derived  intelligence.  (Part  2). 

(10)  Assess  the  efficacy  of  cross  rastered  image  display  in  terms  of 

improved  interpretability  and  image  interpreter  acceptance.  (Part  1 
Stereo,  Measurements). 

Operational  Evaluation  Objectives. 

Objectives  in  this  group  deal  with  the  effectiveness  of  the  CPTB  approach 
to  soft  copy  imagery  exploitation  to  support  USAF  mission  requirements  in 
an  operational  environment. 

(1)  Evaluate  the  effectiveness  of  the  CPTB  approach  to  soft  copy  imagery 
exploitation  to  support  USAF  requirements  for  the  imagery  derived 
intelligence  reports.  (Part  2,  Part  3). 

(2)  Evaluate  the  effectiveness  of  the  CPTB  approach  to  soft  copy  imagery 
exploitation  to  support  the  USAF  requirements  for  H0TPH0T0REP,  IPIR, 
SUPIR,  MIPIR  production.  (Part  2,  Part  3). 

(3)  Evaluate  the  effectiveness  of  the  CPTB  approach  to  soft  copy  imagery 
exploitation  to  support  USAF  requirements  for  new  target  detailed 
descriptions  including  coordinate  determination.  (Part  2)  (Part  1, 
Measurements). 

(4)  Evaluate  the  effectiveness  of  the  CPTB  approach  to  soft  copy  imagery 
exploitation  to  support  USAF  requirements  for  the  production  of  OB 
counts  and  significant  OB  changes.  (Part  2,  Part  3). 

(5)  Evaluate  the  effectiveness  of  CPTB  approach  to  soft  copy  imagery 
exploitation  to  support  USAF  requirements  for  the  extraction  of 
essential  elements  of  information  for  targets  from  relevant  func¬ 
tional  categories.  (Part  2,  Part  3). 
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(6)  Evaluate  the  effectiveness  of  the  CPTB  concept  of  an  on-line  inte¬ 
grated  data  base.  (Part  3). 

(7)  Appraise  CPTB  reliability,  maintainability,  and  supportability  as 
evidenced  by  general  system  availability  throughout  the  test  period 
based  upon  system  contractor  records  of  unscheduled  maintenance 
activities.  (All  tests). 

(8)  Identify  CPTB  personnel  requirements  for  both  system  operators  and 
image  interpreters  skill  levels  and  special  training  for  effective 
system  utilization  in  the  operational  environment.  (All  tests). 

(9)  Assess  the  adequacy  of  CPTB  design  in  the  areas  of  human  factors 
and  safety  as  perceived  by  both  operators  and  image  interpreters. 

(All  tests). 

(10)  Appraise  adequacy  of  CPTB  design  and  documentation  to  facilitate 
software  maintainability  and  operability.  (Entire  test  period). 

(11)  Identify  CPTB  functional  deficiencies  and  specific  system  features/ 
at  tributes/equipments  requiring  improvement.  (All  tests). 

(12)  Identify  specific  CPTB  functional  capabilities  of  features  which  are 
not  required  for  the  accomplishment  of  the  USAF  mission.  (All  tests). 

5.  Evaluation  Criteria 

The  establishment  of  thresholds,  standards,  and  goals  to  be  employed  in 
this  test  is  not  the  straightforward  endeavor  envisioned  in  AFM  55-43. 

The  process  revolves  around  the  notion  that  the  "standards  reflect  the 
operational  requirements  baseline  contained  in  the  system  operational 
and  maintenance  concepts  and  other  appropriate  documentation".  Thresholds 
and  goals  are  then  set  to  bracket  the  standard  so  that  the  "threshold  is 
the  minimum  level  of  acceptable  performance"  and  "...  a  goal  is  a  level 
(of  performance)  that  will  enhance  the  system". 

The  CPTB  system  documentation  does  not  contain  clear  statements  of  perfor¬ 
mance  specifications  for  the  accomplishment  of  interpretation  tasks/subtasks 


using  the  system.  The  system  specification  does  not  stipulate  how  quickly 
or  accurately  an  interpreter  using  the  system  should  be  able  to  complete 
a  report  item,  perform  a  measurement,  search  a  route,  or  accomplish  simi¬ 
lar  activities  which  are  the  basis  for  the  MOE’s.  It  is  also  true  that 
reliable  data  regarding  image  interpreter  performance  of  these  tasks, 
using  current  methods,  is  not  available.  Every  operational  unit  strives 
to  produce  the  highest  quality  reports  in  as  timely  a  fashion  as  possible, 
but,  due  to  the  variability  in  requirements  among  commands  (and  even  within 
a  given  command  from  time  to  time)  no  universal  standards  have  been  dev¬ 
eloped;  no  systematic  performance  records  have  been  kept. 

The  test  approach,  adopted  herein,  utilizes  concurrently  collected  hard 
copy  performance  data  as  the  threshold  (i.e.,  CPTB  threshold  =  average 
hard  copy).  The  absolute  value  of  that  threshold  will  be  determined  by 
the  data  collected  on  the  specific  measures  of  effectiveness  (MOE)  emp¬ 
loyed  in  a  particular  part  of  the  test.  The  standards  in  each  case, 
will  be  defined  as  an  improvement  in  the  average  performance  score  on  the 
relevant  MOE  statistically  significant  at  the  5%  level  of  a  one  tailed 
test.  The  goal  will  be  defined  as  an  improvement  in  average  performance 
statistically  significant  at  greater  than  the  0.5%  level  of  a  one  tailed 
test.  It  is  recognized  that  this  form  of  threshold/standard/goal  does 
not  directly  address  the  absolute  effect  size  (or  performance  improvement) 
required  to  achieve  the  standard  on  any  particular  MOE.  The  absolute  size 
of  the  improvement  required  to  reach  standard  or  goal  will  vary  depending 
on  the  number  of  data  points  obtained  for  a  specific  MOE.  The  important 
point,  however,  is  that  for  any  given  MOE  the  difference  in  effect  size 
between  standard  and  goal  is  a  great  deal  larger  than  that  between  thres¬ 
hold  and  standard. 

Finally,  it  should  be  noted  that  since  the  overall  test  involves  a  large 
number  and  variety  of  MOE’s,  the  overall  evaluation  of  system  effective¬ 
ness  cannot  be  determined  by  failure  to  reach  the  threshold  or  standard 
on  any  single  given  MOE.  An  overall  judgment  of  the  uitlity  of  system 
will  have  to  be  made  on  the  basis  of  the  overall  pattern  of  result,  with 
more  weight  being  given  to  the  more  global  MOE’s  of  report  quality/ 
timeliness  as  compared  to  the  more  specific  MOE’s  such  as  "time  to  achieve 
stereo  model."  In  general,  failure  to  at  least  achieve  threshold 
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performance  on  any  MOE  will  be  considered  by  the  TPWG  (mainly  the  parti¬ 
cipating  command  representatives)  to  assess  the  relevance  for  specific 
applications  of  the  CPTB  approach  to  the  command  requirements. 

6.  Data  Management. 

The  data  to  be  collected  can  be  broadly  calssified  into  four  types; 

(1)  product,  (2)  process,  (3)  activity,  and  (4)  subjective  evaluations. 
Different  forms  of  each  type  will  be  used  in  each  of  the  three  parts  of 
the  test.  The  following  paragraphs  outline  the  data  requirements  by 
test  part  and  for  the  test  activity  as  a  whole. 

A.  Part  1.  This  portion  of  the  test  is,  from  a  data  collection/management 
point  of  view,  the  most  complex  since  it  is  composed  of  four  separate 
subtests  which  are  conducted  for  two  test  interpreters  at  a  time,  in 

an  interleaved  fashion.  This,  coupled  with  the  large  number  of  data 
points  involved,  will  require  careful  control  on  a  continuing  basis. 

The  major  data  for  this  part  of  the  test  will  be  process  type  data. 
Monitors  will  collect  detailed  time  data  for  specific  actions/subtasks. 
Responses  which  can  be  scored  right/wrong  will  be  obtained  for  discreet 
items  (e.g.  measurements,  identifications,  etc.).  Test  interpreters 
will,  for  certain  tasks,  make  confidence  judgments  for  specific  tasks 
(e.g..  Rate  your  confidence  that  particular  EEI’s  could  be  reliably 
discerned).  Test  interpreters  will  also  make  detailed  evaluations  of 
the  utility  of  various  CPTB  features  (.e.g.  types  of  enhancements). 

The  data  will  be  pooled  by  subtest  across  all  test  interpreter  partici¬ 
pants,  maintaining  information  regarding  test  condition,  frame,  collec¬ 
tion  parameters  and  other  relevant  analysis  dimensions. 

B.  Part  2.  The  DIAM  57-7  reporting  task  is  the  basic  organizing  concept 
for  both  test  conduct  and  data  collection.  Test  monitors  will  obtain 
detailed  timing  data  for  specific  steps  in  the  process  for  both  hard 
copy  and  soft  copy  conditions.  Overall  report  preparation  time  will 
also  be  recorded.  Test  interpreters  will  produce  DIAM  57-7  report 
items  which  can  be  scored  for  completeness  and  accuracy  against  the 
school  solution.  End  of  test  questionnaires  will  systematically 


solicit  test  interpreter  evaluations  of  CPTB  utility  in  support  of 
DIAM  57-7  reporting. 

The  process  data  collected  by  the  monitors,  pooled  across  all  test 
interpreters  and  partitioned  by  hard  copy  vs.  soft  copy,  will  provide 
an  empirical  basis  for  evaluating  CPTB  effectiveness  at  each  step  in 
the  reporting  process.  The  product  data,  timeliness,  completeness, 
and  accuracy,  obtained  from  scoring  the  individual  reports  will  be 
similarly  pooled  and  partitioned  and  will  provide  a  reliable  basis 
for  determining  CPTB  contribution  to  mission  performance.  Hard  copy 
annotated  materials  produced  during  the  test  will  be  retained  and 
systematically  evaluated  for  overall  quality  and  usefulness.  Ratings 
from  these  evaluations  will  be  averaged  to  provide  stable  estimates 
of  the  adequacy  of  CPTB  capabilities  provided. 

Raw  data  (monitor  sheets,  reports,  hard  copy,  etc.)  will  be  sorted 
for  test  interpreter  for  each  test  exercise.  These  data  will  be  re¬ 
duced  to  an  unclassified  form  of  key  punch  creation  sheets  for  analy¬ 
sis.  A  backup  copy  of  each  unclassified  data  sheet  will  be  maintained 
in  the  individual  test  interpreter/test  exercise  folder. 

Part  3.  The  emphasis  in  this  portion  of  the  test  will  be  on  product 
data,  supported  by  observer  obtained  activity  data,  and  test  inter¬ 
preter  evaluation.  The  product  data  will  be  derived  from  reports 
produced  under  each  of  the  three  test  conditions  in  two  different 
ways.  The  overall  hard  copy  vs.  soft  copy  quality  comparison  will 
be  made  by  judges  using  a  forced  choice  paired  comparison  procedure. 
Selected  reports  will  be  scored  against  "school  solutions"  derived 
from  appropriate  first,  second,  third  phase  reports.  Hard  copy  rec¬ 
ords  of  all  soft  copy  reports  produced  will  be  filed  in  terms  of  test 
interpreter  and  test  condition  (with/without  PACER  integration).  A 
hard  copy  record  of  the  corresponding  report,  produced  by  current 
methods  by  544th  S1W  personnel  will  be  attached  to  each  CPTB  record. 
Associated  observational  data  will  be  filed  with  each  report  folder, 
maintaining  the  correspondence  necessary  to  associate  production 
time/activities  with  each  individual  output.  Unclassified  data 
records  containing  time,  activity,  and  quality  scores  will  be  prepared 
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for  each  valid  data  point.  (Valid  here  implies  matching  hard  copy/ 
soft  copy  reports;  if  either  are  missing,  the  data  point  is  invalid). 

D.  General.  In  addition  to  those  data  obtained  as  a  direct  result  of 
specific  test  exercises,  a  substantial  amount  of  evaluative  data  is 
also  anticipated.  Included  in  this  category  are  the  following: 

(1)  Training  evaluations.  Each  test  interpreter  will,  at  the  end  of 
each  training  course  (Parts  1,  2,  and  3)  be  required  to  complete 
ATC  form  258  and  a  supplementary  questionnaire. 

(2)  Suitability  evaluation.  Data  regarding  down  time,  system  availa¬ 
bility,  failures,  spare  usage,  and  hardware  problem  reports  will 
be  collected  and  retained  throughout  the  test  period. 

(3)  Software  evaluation.  Software  documention  will  be  appraised 
by  developing/using  command  representatives.  Software  problem 
reports  will  be  completed  and  retained. 

(4)  System  capabilities  (additions/deletions).  All  test  team  members 
will  complete  a  systematic  review  of  the  system  utilization  dur¬ 
ing  test  and  make  recommendations  for  modifications  if  required. 

7.  Responsibilities,  Support,  and  Resources. 

All  parts  of  the  CPTB  evaluation  are  planned  and  controlled  by  a  Test 
Planning  Working  Group  (TPWG)  consisting  of  representatives  from  HQ  USAF, 
AFSC,  RADC,  participating  commands,  AFTEC,  the  system  contractor,  and  the 
test  contractor,  with  advisors  and  observers  designated  from  supporting 
commands  and  other  interested  commands,  services,  and  agencies.  Those 
participants  having  an  active  role  in  the  development  of  the  test,  pre¬ 
paration  of  the  test  materials  and  the  actual  test  conduct  and  reporting 
are  RADC,  SAC,  544th  SIW,  participating  commands  (460th  RTS,  497th  RTG, 
548th  RTG,  and  FTD),  AFTEC,  ATC,  the  test  contractor  (IFR),  and  the  system 
contractor  (NED).  The  following  paragraphs  detail  the  responsibilities 
of  each. 
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A.  RADC/IRRS. 

As  the  Responsible  Test  Organization  (RTO) ,  the  RADC  System  Acquisition 

Office  (RADC/IRRS)  shall: 

(1)  Appoint  a  Test  Director  (TD)  fo  the  CPTB  Test  Program. 

(2)  Define  Test  Planning  Working  Group  (TPWG)  organizational  relation¬ 
ships  and  responsibilities. 

(3)  Develop  and  plan  the  T&E  program  to  meet  the  program  decision 
milestones . 

(4)  Work  with  TPWG  to  identify  critical  questions  and  areas  of  risk 
to  be  addressed  as  test  objectives  during  T&E  phases. 

(5)  Provide  for  contractor  participation  in  DT&E  and  operational 
evaluation  as  required. 

(6)  Direct,  control,  review,  and  approve  contractor  generated  test 
plans,  procedures,  test  data  analyses,  and  test  reports  for  DT&E. 

(7)  Coordinate  contractor’s  DT&E  test  plan  and  test  reports  with 
appropriate  agencies. 

(8)  Provide  or  facilitate  acquisition  of  test  monitors,  test  material/ 
data  control  librarian,  and  other  test  support  personnel. 

B.  Hq  SAC. 

SAC/INCR  will  be  the  executive  or  lead  supporting  organization  and 

shall : 

(1)  Review  and  approve  the  operational  evaluation  plans  and  procedures. 

(2)  Act  as  the  single  SAC  point  of  contact  for  the  CPTB. 

(3)  Act  as  the  T&E  program  interface  with  the  544th  SIW  for  system 
requirements. 
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C. 


(4)  Coordinate  facility  requirements  for  the  CPTB. 

(5)  Be  responsible  for  all  security  requirements. 


t 

i 


SAC/ AD  will  be  responsible  for  the  CPTB  interface  with  PACER  and  will 
specifically : 

(1)  Coordinate  PACER  testing  requirements  with  respect  to  CPTB. 

(2)  Coordinate  PACER  hardware/software  interface  with  CPTB. 


(3)  Coordinate  TEMPEST  testing  for  the  CPTB  at  Hq  SAC. 

(4)  Provide  data  base  support  for  the  test  program. 

(5)  Provide  operations  support  (e.g.,  system  operators,  tapes,  etc.). 

544th  SIW. 

The  544th  SIW  as  the  host  organization  shall: 

(1)  Provide  guidance  to  the  TPWG  in  developing  test  objectives  and 
procedures  which  accurately  test  the  photo  interpreter/CPTB 
interface . 

(2)  Provide  a  representative  to  the  TPWG  who  will  act  as  Operational 
Test  Director. 

(3)  Assist  in  the  selection  and  preparation  of  imagery /test  material 
for  Part  1  tests. 

(4)  Provide  working  space  for  test  contractor  personnel.  ( 

(5X  Select  imagery  for  Part  2  testing. 

i 

(6)  Provide  two  photo  interpreters  to  assist  in  test  material 
preparation. 
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(7)  Provide  eight  test  interpreters  (2  hours/day  each)  for  Part  2 
test ing. 

(8)  Provide  test  interpreters;  three  for  Part  1  and  five  for  Part  3 
testing. 

(9)  Provide  photo  lab  support  for  test  material  preparation. 

(10)  Provide  for  reproduction  of  classified  test  materials. 

Participating  Commands. 

The  commands  falling  in  this  group  (460th  RTS,  497th  RTG,  548th  RTG, 

and  FID)  shall: 

(1)  Appoint  a  regular  representat  ive  to  the  TPWG. 

(2)  Provide  guidance  to  TPWG  to  ensure  that  testing  incorporates 
exercises  and  performance  measures  relevant  to  the  mission  of 
other  participating  commands. 

(3)  Provide  at  least  one  test  interpreter  for  two  weeks  at  SAC  for 
Part  1  testing. 

(4)  Provide  at  least  one  test  interpreter  for  four  weeks  at  SAC 
for  Part  3  testing. 

(5)  Coordinate  on  all  test  documentation. 

Air  Force  Test  and  Evaluation  Center  (AFTEC) . 

AFTEC  shall: 

(1)  Appoint  a  representative  to  the  TPWG. 

(2)  Review  and  approve  the  Operational  plans  and  procedures. 

(3)  Monitor  the  Operational  Evaluation  as  required. 
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(4)  Review  and  comment  on  the  Operational  Evaluation  report,  as 
appropriate . 

F.  Air  Training  Command  (ATC/TCHTG) . 

ATC  shall: 

(1)  Provide  a  representative  to  the  TPWG  who  shall  coordinate  train¬ 
ing  development/evaluation. 

(2)  Provide  one  instructor  to  develop  and  conduct  specialized  train¬ 
ing  for  test  interpreters  prior  to  participation  in  each  Part  of 
the  test, 

G.  Northrop  Electronics  Division  (NED). 

Northrop  Electronics  Division  as  system  contractor  shall: 

(1)  Develop  test  materials  and  procedures  for  acceptance  testing  to 
be  performed  at  the  contractor’s  facility  and  at  SAC. 

(2)  Perform  acceptance  testing  based  upon  contractor  submitted  and 
government  approved  test  plan. 

(3)  Reduce  and  analyze  test  data;  prepare  and  provide  test  reports 
to  Program  Office  for  review. 

(4)  Provide  all  maintenance  to  the  CPTB  during  acceptance  testing  and 
combined  DT&E/Operational  Evaluation  except  for  government  pro¬ 
vided  maintenance  as  identified  in  the  contract. 

(5)  Provide  supply  support  for  all  testing  conducted  at  the  contrac¬ 
tor’s  facility.  For  those  test  phases  at  Offutt,  AFB  provide 
supply  support  for  all  CPTB  equipment  with  the  exception  of  those 
items  identified  in  the  contract. 

(6)  Provide  training  for  ten  individuals  from  the  test  team  including 
system  operators.  Test  Director,  Test  Manager,  photo  interpreters, 
and  ATC  instructor. 
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H.  Institute  For  Research  (IFR). 

The  Institute  For  Research  as  the  test  contractor  with  assistance 

from  Pattern  Analysis  and  Recognition  Corporation  (PARC)  and  Science 

Applications  Incorporated  (SAI)  shall: 

(1)  Generate  a  formal  test  plan  and  procedures  for  the  combined  DT&E/ 
Operational  Evaluation  as  defined  in  RADC  Contract  F30602-78-C-0072 
and  a  follow-up  contract  (F30602-80-C-0205) . 

(2)  Effect  close  and  extensive  coordination  with  participating  com¬ 
mands  to  develop  test  objectives;  determine  material,  personnel, 
and  scheduling/ team  requirements  of  the  test,  and  define  opera¬ 
tional  scenarios. 

(3)  Provide  test  support  consisting  of  an  on-site  test  manager  and 
manpower  for  monitoring  the  test,  test  material  preparation,  and 
reducing  test  data  into  the  form  of  an  interim  and  final  reports. 


SECTION  III.  TEST  REPORTS  AND  BRIEFINGS 


1.  Status  Reports. 

Because  of  the  nature  and  length  of  the  test  program  it  has  been  deter¬ 
mined  that  interim  reporting  of  P<^rt  1  and  Part  2  results  would  be  de¬ 
sirable.  Periodic  status  reports  (monthly)  are  required  to  help  identify 
potential  problem  areas  early  in  testing  and  to  provide  an  indication  of 
the  systems  capabilities.  The  Interim  Reports  for  Part  1  and  Part  2 
will  be  prepared  by  the  test  contractor  and  submitted  to  the  TPWG 
for  review  and  comment.  The  reports  will  be  limited  to:  (1)  a  brief 
description  of  the  test  conduct  including  any  deviations  from  procedures, 

(2)  summary  and  discussion  of  major  results  organized  by  subtest/test, 
and  (3)  preliminary  conclusions  and  recommendations  relative  to  those 
test  objectives  addressed  by  that  portion  of  the  test.  The  format  will 
be  contractor  developed  subject  to  TPWG  approval  and  designed  to  facil¬ 
itate  timely  reporting  upon  completion  of  the  subject  test  portion. 

2.  Deficiency  Reports. 

While  the  CPTB  does  not  meet  the  requirements  given  in  T.O.  00-35D-54, 

Section  V  for  full-scale  service  reporting,  many  aspects  of  the  program 
make  formal  reporting  and  review  of  suspected  deficiencies  and  proposed 
enhancements  very  beneficial.  Any  suspected  deficiencies  or  proposed 
enhancements  discovered  by  the  test  team  will  be  recorded  on  the  AFTO 
Form  240  in  accordance  with  the  procedures  given  in  paragraphs  5-5, 

T.O.  00-35D-54.  The  reports  will  be  forwarded  to  RADC/IRRS  for  review 
and  comment,  as  appropriate.  This  formalized  procedure  will  provide 
background  information  for  development  of  any  follow-on  system. 

3.  Final  Test  Report. 

The  Final  Test  Report  consisting  of  three  volumes  (Overview,  DT&E, 

Operational  Evaluation)  modeled  after  the  suggested  format  from  AFM 

55-43  will  be  submitted  for  review  and  comment  to  the  TPWG.  Following 

revisions  resulting  from  this  review  the  DT&E  volume  will  be  submitted 

for  approval  to  RADC  and  AFSC;  the  Operational  Evaluation  volume  will 

be  submitted  through  HqSAC  to  HqAFTEC  for  evaluation  and  comment  to  ^ 

HqUSAF. 


J 
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4,  Briefings. 

As  a  non-major  program  there  are  no  required  briefings.  Briefings  of 
progress  to  date  will  be  made  by  test  team  personnel  to  the  TPWG  at 
scheduled  meetings  throughout  the  test  period.  Additional  briefings 
for  Hq  SAC,  USAF/INY,  AFSC  or  other  program  management  elements  will 
be  prepared  and  conducted  as  required. 

A  video  taped  test  summary  and  system  demonstration  will  be  prepared, 
under  SAC  auspices,  by  a  U.S.  Army  Unit  authorized  to  accomplish  this 
type  of  recording  at  the  required  classification/caveat  levels  involved 
in  this  program.  Test  team  members  will  support  this  activity  on  a  not- 
to-interfere  basis. 
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ANNEX  A 

OPERATIONAL  EFFECTIVENESS 
AND 

CONCEPT  VALIDATION 
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A-l  Part  1:  Soft  copy  capabilities  testing. 

Part  1  of  the  test  will  consist  of  four  special  tests:  (1)  Stereo, 

(2)  Subpictures,  (3)  Measurements,  and  (4)  Enhancements.  Each  is 
described  separately  in  the  following  paragraphs. 

A-l-1  Stereo  test. 

A.  General. 

This  test  is  designed  to  investigate  the  utility,  effectiveness  and 
acceptability  of  the  CPTB  capability  for  up  to  full  frame  stereo  view¬ 
ing.  The  design  permits  assessment  of  relative  object  size,  separation 
and  location  and  human  factors  considerations. 

B.  Standard  of  performance. 

No  absolute  standard  exists.  Data  will  be  concurrently  collected  to 
establish  a  hard  copy  baseline  for  comparison.  Information  regarding 
ground/image  truth  will  be  required. 

C.  Measures  of  Effectiveness.  (MOE). 

(1)  Time:  Elapsed  time  data  for  individual  and  total  exercise  will 
be  recorded. 

(2)  Correctness:  Responses  to  specific  questions  (e.g.  which  is 
higher,  A  or  B)  will  be  recorded  and  compared  to  ground/ image 
truth. 

(3)  Acceptance:  Test  interpreters  will  be  systematically  queried 
regarding  advantages/disadvantages  of  unaided  stereo  veiwing. 

D.  Test  preparation. 

(1)  Imagery  Selection:  Imagery  selection  will  be  accomplished  by 
RADC/IFR.  Twenty-four  sets  of  stereo  images  will  be  required 
having  characteristics  which  minimize  extraneous  height  cues. 

The  set  will  be  a  mixture  of  operational  and  controlled  range 


scenes. 
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(2)  Image  preparation:  For  each  image  a  number  of  objects  or  features 
will  be  selected,  systematically  varying  relative  height,  separation 
and  frame  location.  The  selected  ob jects/f eatures  will  be  annotated 
and  specific  questions  formulated.  Note:  initial  work  completed 
using  hard  copy  validated  and  transferred  to  soft  copy  during  Pre¬ 
test  Preparation. 

(3)  Instructions/Data  Collection:  Imagery  will  be  subdivided  into  test 
exercises,  6  stereo  pairs  per  exercise,  and  specific  detailed  instru¬ 
ctions  for  both  hard  copy  and  soft  copy  conditions  written  for  each 
exercise.  The  data  collection  forms  for  recording  task  time  and 
test  interpreter  responses  will  be  prepared  for  each  pair. 

(4)  Data  Base:  No  special  requirement,  except  for  the  CPTB  frame 
Initialization . 

E.  Test  Conduct 

(1)  Schedule:  Each  test  interpreter  will  be  required  to  complete  four 
sessions  of  2  hours  each.  Two  sessions  will  be  on  the  CPTB  station; 

2  at  a  hard  copy  light  table.  Sessions  will  be  interleaved  with 
other  sessions  required  for  the  other  Part  1  tests.  Master  sched¬ 
ule  is  shown  in  A-l-5. 

(2)  Test  Personnel:  This  test  will  require  10  test  interpreters  for  8 
hours  each.  Two  test  monitors  will  be  required;  one  each  for  the 
soft  copy  and  hard  copy  stations.  Also  required  will  be  a  system 
operator  and  a  test  materials/data  collection  control  clerk. 

F.  Data  Reduction/Analysis 

Data  collection  forms  will  be  scored/ tabulated ,  as  received,  and  recorded 
on  coded  keypunch  creation  sheets.  (Since  it  is  anticipated  that  this 
test  wil L  produce  approximately  3,400  data  points,  computer  processing 
is  required).  Time  and  correctness  scores  will  be  separately  aggre¬ 
gated  accross  individuals  and  images  and  analyzed  by  means  of  three 
way  analysis  of  variance  (ANOVA). 
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A-l-2  Subpictures  test. 

A.  General. 

This  test  is  designed  to  investigate  the  utility  and  effectiveness  of 
the  various  types  and  purposes  of  CPTB  subpictures  capability.  Of  parti¬ 
cular  interest  will  be  optimum  presentation  and  search  tasks  using  both 
overview  reference  and  subpictures  took  ahead. 

B.  Standard  of  Performance. 

No  absolute  standard  exists.  Data  will  be  concurrently  collected  to 
establish  a  hard  copy  baseline  for  comparison. 

C.  Measures  of  effectiveness.  (MOE) . 

(1)  Time:  Elapsed  time  data  for  individual  tasks  (e.g.  time  to  locate 
target,  time  to  complete  search)  will  be  recorded. 

(2)  Correctness:  Responses  to  specific  questions  (e.g.  Is  the  vehicle 
at  point  A  tracked  or  wheeled?)  will  be  recorded  and  scored  against 
ground  (image)  truth. 

(3)  Completeness:  Responses  to  search/count ing  tasks  will  be  recorded 
and  scored  against  the  school  solution  and  percent  complete  calcu¬ 
lated  where  %  C  -  Nbr.  found/Nbr.  present. 

(4)  Accuracy:  Responses  to  search/counting  tasks  which  also  require 
identification  of  objects  will  be  scored  for  accuracy  where 

%  A  =  Nbr.  correctly  ident if ied/Nbr .  detected. 

D.  Test  Preparation. 

(1)  Imagery  selection:  Imagery  selection  will  be  accomplished  by 

RADC/IFR.  Thirty-six  images  will  be  required  and  will  be  selected 
on  the  basis  of  scene/target  relationship  and  image  format.  The 
set  will  contain  a  variety  of  the  major  acquisition  parameters  of 
slant  range,  dynamic  range,  and  sun  angle. 
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(2)  linage  preparation;  For  each  image  specific  tasks  will  be  developed 
involving  locating  targets  of  various  types  (e.g.  small  target 
imbedded  in  complex  area,  specific  features  of  larger  targets), 
area,  and  route  searches,  etc.  School  Solutions  derived  from 
ground  truth  or  image  truth  will  be  developed  and  validated  in  both 
bard  copy  and  soft  copy  form. 

(3)  Instructions/Data  Collection:  Imagery  will  be  subdivided  into  test 
exercises,  6  images  per  exercise.  Specific  monitor  and  test  analyst 
instructions  will  be  written  for  each  exercise  for  both  hard  copy 
and  soft  copy  conditions.  Instructions  will  control  type  and  se¬ 
quence  of  subpictures  options  used  and  specific  tasks  to  be  accom¬ 
plished.  Associated  with  the  specific  instructions  for  the  monitor 
will  be  data  to  be  collected  (start/stop  times,  target  detections/ 
identifications,  counts,  etc.). 

(4)  Data  Base:  For  those  targets  where  the  optimum  presentation  and/or 
target  symbols  options  are  to  be  employed,  it  will  be  necessary  to 
insure  that  the  PACER  data  base  contains  the  requisite  information. 

E.  Test  Conduct. 

(1)  Schedule:  Each  test  interpreter  will  be  required  to  complete  six 
sessions  of  two  hours  each.  Three  sessions  will  be  on  the  CPTB  sta¬ 
tion;  3  at  a  hard  copy  light  table.  Each  interpreter  will  see  all 
of  the  imagery.  Viewing  will  be  balanced  across  interpreters  by 
hard  copy  vs.  soft  copy.  Test  sessions  will  be  interleaved  with 
other  sessions  required  for  other  Part  1  tests.  Master  schedule 

is  shown  in  Section  A-l-5. 

(2)  Test  Personnel:  This  test  will  require  10  test  interpreters  for  12 
hours  each.  Two  test  monitors  will  be  required;  one  each  for  the 
soft  copy  and  hard  copy  stations.  Also  required  will  be  a  system 
operator  and  test  materials/data  collection  control  clerk. 
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F.  Data  Reduct  ion/ Analysis . 

Data  collection  forms  will  be  scored/ tabulated  as  received  and  recorded 
on  coded  key  punch  creation  sheets.  Approximately  7500  data  points  are 
anticipated.  Scores  on  all  MOE's  will  be  aggregated  across  interpreters 
by  specific  image  and  test  condition  and  analyzed  by  means  of  standard 
ANOVA  routines. 

-1-3  Measurements  Test. 

A.  General. 

This  test  is  designed  to  investigate  the  accuracy  of  measurement  achiev¬ 
able  using  CPTB.  Measurement  accuracy  and  coordinate  determination  will 
be  assessed  in  combination  with  other  CPTB  capabilities  for  geometric  cor¬ 
rection,  coverage  correction,  and  stereo  viewing.  Location,  object  size, 
image  size,  and  frame  orientation  will  be  varied. 

B.  Standard  of  Performance. 

Measurement  accuracy  will  be  determined  in  comparison  to  ground  truth  as 
modified  by  range  of  expected  accuracy  as  predicted  for  individual  test 
frames . 

C.  Measures  of  Effectiveness.  (MOE) . 

(1)  Dimension  accuracy:  Lengths,  widths,  heights,  areas,  volumes,  and 
azimuth  will  he  recorded  and  absolute  and  percent  deviation  from 
ground  truth  calculated. 

(2)  Coordinate  accuracy:  Absolute  deviation  from  known  ground  truth 
location  will  be  recorded. 

(3)  Cursor  positioning:  Test  interpreters  will  be  systematically  queried 
regarding  cursor  positioning  via  sonic  pen  and  joy  stick. 

(4)  Time:  Elapsed  time  data  for  individual  and  total  exercise  will  be 
recorded . 
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D.  Test  Preparation. 

(1)  Imagery  selection:  Imagery  selection  will  be  accomplished  by  RADC/ 
IFR.  Twelve  images  will  be  required,  representing  selected  levels 
of  geometric  and  coverage  distortion  and  a  wide  range  of  objects/ 
features  to  be  measured.  The  set  will  consist  of  controlled  range 
scenes . 

(2)  Image  preparation:  For  each  image  approximately  20  objects/features/ 
dimensions  will  be  selected  and  annotated,  systematically  varying  loc¬ 
ation,  size  orientation,  etc.  Specific  features  of  known  location 
will  be  identified  for  coordinate  determination  tasks.  Initial  work 
will  be  accomplished  on  hard  copy,  later  validated  and  transferred 

to  soft  copy. 

(3)  Instructions/Data  Collection:  Imagery  will  be  subdivided  into  test 
exercises,  6  images  per  exercise,  and  specific  monitor  instruction/ 
data  collection  forms  constructed  for  each  exercise.  Test  inter¬ 
preter  instructions  will  also  be  written. 

(4)  Data  Base:  It  will  be  necessary  to  prepare  and  insert  into  test 
PACER  information  needed  to  support  coordinate  determination  tasks. 

E.  Test  Conduct. 

(1)  Schedule:  Each  test  interpreter  will  be  required  to  complete  two 
test  sessions  of  two  hours  each,  both  on  the  CPTB  station.  Sessions 
will  be  interleaved  with  other  sessions  required  for  other  Part  1 
test.  Master  schedule  is  shown  in  Section  A-l-5. 

(2)  Test  Personnel:  This  test  will  require  10  test  interpreters  for 
four  hours  each.  One  test  monitor  will  be  required  along  with  one 
system  operator  and  the  test  materials/data  collection  control  clerk. 


F.  Data  Reduction/Analysis. 

Data  collection  forms  will  be  scored  against  ground  truth  and  recorded 
on  key  punch  creation  sheets.  It  is  anticipated  t ha t  approximately 
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3400  data  points  will  result  from  this  test.  The  data  will  be  analyzed 
by  means  of  standard  ANOVA  routines  and  results  produced  for  total  ac¬ 
curacy  and  accuracy  as  a  function  of  object  size,  corrections  employed, 
and  measurement  option  exercised. 

A- 1-4  Enhancement  test. 

A.  General. 

This  test  is  designed  to  evaluate  in  a  demonstration  mode,  the  CPTB  capa¬ 
bilities  for  image  enhancement  and  their  potential  utility  in  increasing 
the  level  of  detail  and  interpreter  confidence  achievable  in  the  exploi¬ 
tation  of  soft  copy  imagery. 

B.  Standard  of  Performance. 

No  absolute  standard  exists.  Comparison  will  be  made  between  enhanced 
and  unenhanced  views  of  the  same  scene. 

C.  Measures  of  Effectiveness.  (MOE) . 

(1)  Information  gain:  Difference  scores  calculated  between  enhanced 
and  unenhanced  views. 

(3)  Confidence  gain:  Shifts  in  interpretation  categories  (possible/ 
probabl e/conf irmed)  will  be  recorded. 

D.  Test  Preparation. 

(1)  Imagery  selection:  Imagery  selection  will  be  accomplished  by  RADC/ 
IFR.  Twenty-four  images  will  be  required  representing  a  range  of 
conditions  and  scenes  judged  to  be  appropriate  for  the  various  en¬ 
hancement  capabilities  of  CPTB.  The  image  set  will  be  drawn  from 
operational  imagery  available  at  SAC.  All  selection  will  be  accom¬ 
plished  on  the  CPTB  station. 

(2)  Image  preparation:  For  each  image,  objects  will  be  selected  and 
annotated  and  an  enhancement  option  determined.  A  subpicture  with 
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the  selected  enhancement  will  be  prepared  and  saved.  School  solu¬ 
tions  for  each  object  will  be  determined. 

(3)  Instruction/data  collection:  Specific  instruction  sets  for  image, 
including  a  detailed  enhancement  script  will  be  written.  Data  col¬ 
lection  forms  for  object  interpretation,  confidence  level,  and  inter¬ 
preter  rating  will  be  prepared  corresponding  to  each  object.  Test 
interpreter  and  monitor  instructions  for  the  2  hours  test  session 
will  also  be  prepared. 

E.  Test  Conduct. 

(1)  Schedule:  Each  test  interpreter  will  be  required  to  complete  one 
session  of  two  hours.  The  session  will  be  scheduled  at  a  convenient 
time  during  the  conduct  of  the  other  Part  1  tests.  Master  schedule 
is  shown  in  Section  A- 1-5. 

(2)  Personnel:  The  ten  test  interpreters  utilized  in  the  other  Part  1 
test  will  be  required  to  complete  this  test  as  well.  In  addition, 
the  test/demonstration  is  to  be  structured  so  that  additional  (non 
CPTB  trained  interpreters)  can  act  as  test  interpreters  and  provide 
useful  data.  The  number  of  additional  interpreters  will  be  limited 
by  availability  and  CPTB  scheduleing.  Each  session  will  require  1 
monitor,  1  operator,  and  the  test  materials/data  collection  control 
clerk. 

F.  Data  Reduction/Analysis. 

Monitor  data  collection  forms  will  be  tabulated  as  recieved  and  recorded 
on  key  punch  creation  sheets.  The  quantity  of  data  is  indeterminate  de¬ 
pending  on  the  number  of  additional  test  interpreters  obtained.  Analysis 
will  consist  of  summary  statistics  (average,  variance,  range,  etc.)  across 
interpreters/images  for  each  measure. 

A-l-5  Part  1:  Master  Schedule 

The  following  schedule  is  repeated  four  times  to  accommodate  a  total  of 
eight  test  interpreters. 


DAY  I NT E UP 


1 


3 


4 
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Part  1  Master  Schedule 

Test  Period 


A 

Subp i c/CP  I 

fleas /CPI 

Subpic/HCl 

open 

B 

Subpic/HCl 

open 

Subpic/CPl 

open 

A 

St ereo/HC i 

open 

Stereo/CPI 

open 

B 

Stereo/ CPI 

Meas/CPl 

Stereo/HCL 

open 

A 

Subpic/CP2 

Meas/CP2 

Subpic/HC2 

open 

B 

Subpic/HC2 

open 

Subpic/CP2 

Enhance 

A 

Stereo/HC2 

open 

Sterco/CP2 

open 

B 

Stereo/CP2 

Meas/CP2 

Stereo/HC2 

open 

A 

Subp ic/ CP 3 

Enhance 

Subpic/HC3 

Quest  ion 

B 

Subpic/HC3 

open 

Subpic/CP3 

Question 

A-2  Part  2:  DIAM  57-5  Reporting. 

A.  General. 

Part  2  of  the  test  will  consist  of  a  systematically  structured,  mission 
level  comparison,  utilizing  selected  imagery,  a  controlled  data  base  and 
encompassing  the  entire  exploitation  cycle  from  frame  selection  thru  the 
production  of  report  items  and  the  preparation  of  hard  copy  target  mater¬ 
ials  . 

B.  Performance  Standard. 

No  absolute  standard  exists.  Data  will  be  concurrently  collected  to 
establish  a  hard  copy  baseline  for  comparison, 

C.  Measures  of  Effectiveness.  (MOE) . 

(1)  Time:  Elapsed  time  will  be  recorded  for  all  defined  steps  in  the 
exploitation  cycle  (definitions  matched  for  hard  and  soft  copy), 
for  total  task  time,  per  report  item,  and  for  total  cycle  time  from 
receipt  of  image. 
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(2)  Completeness:  Each  report  item  will  be  scored  against  the  school 
solution  and  a  percent  completeness  score  computed  where  X  C  =  EEI 
responses/EII  required. 

(3)  Accuracy:  For  each  report  item  the  EEI  responses  present  will  be 
scored  against  the  school  solution  and  a  percent  accuracy  score 
computed  where  %  A  =  EEI  correct/EET  present. 

(4)  HC  Quality:  The  required  HC  target  materials  generated  for  each 
report  item  will  be  scored  for  completeness  and  accuracy  and  judged 
for  overall  quality. 

D.  Test  Preparation. 

(1)  Imagery  selection:  Imagery  selection  will  be  accomplished  by  544th 
SIW  assisted,  if  necessary,  by  IFR  and/or  personnel  from  participating 
commands.  A  total  of  56  images  will  be  required,  eight  representing 
each  of  seven  functional  areas.  The  total  set  will  contain  a  range 

of  acquisition  parameters,  wide  geographic  dispersion,  and  approxi- 
mately  15  percent  stereo  coverage. 

(2)  Image  preparation:  The  selected  imagery  will  be  reveiwed  on  the  CPTB 
station  and  the  interrelation  with  the  data  base  and  correspondence 
of  the  school  solution  validated.  Special  monitor  notes  necessary 
for  presentation  control  will  be  developed. 

(3)  Inst ructions/Data  Collection:  The  imagery  will  be  subdivided  into 
28  sessions  consisting  of  2  frames  and  4  targets  each.  Two  targets 
will  be  updated  and  two  will  require  detailed  descriptions.  Monitor 
instructions  and  associated  data  collection  materials  defining  start/ 
stop  times  and  test  control  information  will  be  prepared  for  each 
session  in  both  hard  copy  and  soft  copy  forms. 

(4)  Data  Base:  The  PACER  data  base  corresponding  to  the  image  selection 
period  will  be  frozen.  Records  associated  with  selected  targets  will 
be  modified  to  produce  desired  test  conditions.  All  entries  made  on 
the  basis  of  selected  frames  will  be  deleted  and  saved  as  a  basis 
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for  school  solutions.  Additional  school  solution  material  will  be 
obtained  (wherever  possible)  from  appropriate  1st,  2nd,  and  3rd  phase 
reports  on  covered  targets. 

E.  Test  Conduct. 

(1)  Schedule:  Each  interpreter  will  he  required  to  complete  28  sessions 
of  2  hours  each  during  the  six  week  period  allocated  for  this  part 

of  the  test.  Fourteen  of  these  sessions  will  be  on  the  CPTB  station. 
14  on  the  hard  copy  light  table.  There  will  be  eight  sessions  run 
per  day,  4  each,  hard  and  soft  copy.  Because  of  limitations  on  the 
access  to  TEST  PACER  all  hard  copy  sessions  will  be  run  simultane¬ 
ously  during  the  two  hour  period  when  TEST  PACER  is  available. 

CPTB  sessions  will  be  run  throughout  the  working  day.  Each  Test 
interpreter  will  work  two  hours  each  day. 

(2)  Personnel:  Because  of  the  test  schedule  constraints  of  two  hours 
per  test  interpreter  per  day,  it  appears  most  efficient  to  plan  that 
all  eight  required  test  interpreters  be  drawn  from  the  544th  SIW. 

Also  because  of  TEST  PACER  schedule  constraints  this  test  will  re¬ 
quire  extra  monitors;  a  total  of  5  required  for  one,  two  hour  period. 
The  remainder  of  each  day  will  require  one  monitor,  one  system 
operator,  and  the  test  mater ials /data  collection  clerk. 

F,  Data  Reduction/Analysis. 

Time  data  collection  by  the  test  monitors  will  be  recorded  on  keypunch 
creation  sheets  for  later  processing.  Report  items  produced  will  be 
scored  for  completeness  and  accuracy  against  the  school  solution.  Hard 
copy  target  materials  produced  will  be  scored  for  comple teness/correc tness 
and  retained  for  quality  judging.  Time  scores  will  be  aggregated  across 
interpreters  by  frame  and  report  time  and  statistical  comparisons  of  aver¬ 
ages  for  hard  and  soft  copy  accomplished.  Statistical  tests  (t-test) 
will  also  be  performed  on  completeness  and  accuracy  measures. 
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A-3  Part  3:  Soft  Copy  Exploitation  Concepts 

A.  General. 

Part  3  of  this  test  is  designed  to  accomplish  two  major  purposes: 

(1)  investigate  the  contribution  of  the  CPTB  integrated  data  base  approach 
to  soft  copy  exploitation,  and  (2)  to  provide  intensive  hands-on  exper¬ 
ience  for  interpreters  for  other  using  commands  in  an  operational  environ¬ 
ment.  To  accomplish  this  the  test  will  utilize  operational  imagery  and 
the  actual  PACER  data  base.  The  basis  for  evaluation  will  be  a  threeway 
comparison:  (1)  Hard  copy,  (2)  CPTB  as  designed,  (3)  CPTB  minus  PACER 

integration. 

B.  Standard  of  Performance. 

No  absolute  standard  exists.  Data  will  be  concurrently  collected  to 
establish  a  hard  copy  (current  practice)  baseline  for  comparison  with 
both  CPTB  employment  concepts. 

C.  Measures  of  Effectiveness.  (MOE) . 

(1)  Time:  Records  will  be  kept  regarding  all  milestone  times  from  image 
available  to  report  item  complete  for  all  three  test  conditions. 

(2)  Report  item  quality:  Report  items  will  be  judged  (paired  comparison) 
for  overall  quality  (i.e.  each  report  item  will  be  completed  twice, 
once  in  one  of  the  CPTB  conditions  and  once  in  hard  copy  during  the 
regular  course  of  work  in  the  544th  SIW.  These  two  versions  will  be 
compared  side  by  side  in  a  forced  choice  paired  comparison  made  by 
qualified  USAF  personnel). 

(3)  Completeness:  For  a  subsample  of  the  reports  it  is  anticipated  that 
appropriate  1st,  2nd,  and  3rd  reports  from  other  sources  will  be 
obtainable.  These  will  be  used  as  a  post  hoc  school  solution  that 
will  permit  completeness  score  computation  by  the  standard  means. 

(4)  Accuracy:  In  the  same  fashion  as  above,  accuracy  scores  will  be 
computed  for  a  subset  of  the  total  reports. 
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(5)  Ratings:  All  interpreters  will  complete  extensive  questionnaires 
regarding  their  evaluation  of  the  two  CPTB  employment  conditions. 

D.  Test  Preparation. 

(1)  Imagery  selection:  There  will  be  NO  preselection  of  imagery  except 
to  insure  that  on  each  test  day  the  CPTB  image  queue  contains  suffic¬ 
ient  appropriate  imagery  for  the  assigned  test  interpreter. 

(2)  Image  preparation:  No  special  preparation  of  images  is  required 
except  to  insure  that  on  each  test  day  all  imagery  previously  worked 
on  the  CPTB  station  has  a  Mred  tag"  attached  prior  to  distribution 
on  the  production  floor  of  the  544th  SIW. 

(3)  Instructions/Data  Collection:  Observer  forms  and  image  milestone 
time  logs  will  be  prepared  prior  to  the  start  of  test  to  support 
observer  activity  at  both  CPTB  and  hard  copy  stations  and  to  facili¬ 
tate  record  keeping  of  required  image/report  times. 

(4)  Data  base:  No  special  data  base  preparation  is  required  except  to 
insure  that  CPTB  generated  reports  DO  NOT  actually  enter  the  PACER 
data  base.  (It  will  be  necessary  to  retain  identical  format  copies 
of  both  hard  copy  and  CPTB  produced  reports  for  later  comparison) . 

E.  Test  Conduct. 

(1)  Schedule:  Each  test  interpreter  will  work  for  two  weeks  (five  work 
days,  regular  shift)  one  week  on  CPTB  as  designed,  one  week  on  CPTB 
without  PACER  integration.  Each  test  interpreter  will  receive  two 
weeks  of  CPTB  training  and  practice  (training  shift)  immediately 
prior  to  his  two  week  test  time.  Scheduling  of  time  periods  for 
individual  test  interpreters  will  be  worked  out  at  the  convenience 
of  the  participating  commands.  The  total  test  time  required  will 
be  22  weeks. 

(2)  Personnel:  This  part  of  the  test  will  require  10  test  interpreters, 

2  each  from  the  460th  RTS,  497th  RTG ,  548th  RTS,  and  FTD,  544th  SIW. 
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Also  required  will  be  one  full  time  CPTB  observer  and  three  or  four 
part  time  hard  copy  condition  observers.  Three  qualified  USAF  judges 
will  be  required  periodically  throughout  the  test  period.  CPTB  sys¬ 
tem  operators,  CPTB  instructors,  NED  maintenance  personnel,  and  the 
test  material/data  collection  control  clerk  will  also  be  required. 

F.  Data  Reduction/Analysis. 

All  time  data  will  be  tabulated  as  received.  Report  copies  will  be 
accumulated  and  judged  on  a  biweekly  basis  (i.e.  at  the  completion  of 
each  test  interpreter  period).  Report  grading  against  material  from 
other  sources  will  be  accomplished  at  the  end  of  the  test  period. 
Statistical  comparison  of  CPTB  vs.  CPTB-PACER,  HC  vs.  CPTB,  HC  vs. 
CPTB-PACER  will  be  accomplished.  Approximately  200  data  points 
(reports)  per  interpreter  per  test  condition  are  anticipated. 

A-4  Detailed  Test  Schedule 


Activity  #  Weeks  Start  Date  End  Date 


Pretest  Preparation 

16 

1 

OCT 

80 

16 

JAN 

81 

Dry  Run 

4 

19 

JAN 

81 

13 

FEB 

81 

Part  1  Tests 

6 

9 

FEB 

81 

27 

MAR 

81 

*PC1  Training 

2 

9 

FEB 

81 

13 

FEB 

81 

Test  Run  1 

16 

FEB 

81 

20 

FEB 

81 

PC2  Training 

2 

16 

FEB 

81 

20 

FEB 

81 

Test  Run  2 

2 

MAR 

81 

6 

MAR 

81 

PC3  Training 

2 

2 

MAR 

81 

6 

MAR 

81 

Test  Run  3 

9 

MAR 

81 

13 

MAR 

81 

PC4  Training 

2 

9 

MAR 

81 

13 

MAR 

81 

Test  Run  4 

16 

MAR 

81 

20 

MAR 

81 

PC5  Training 

2 

16 

MAR 

81 

20 

MAR 

81 

Test  Run  5 

23 

MAR 

81 

27 

MAR 

81 

Part  2  Test 

8 

30 

MAR 

81 

22 

MAY 

81 

Part  2  Training 

2 

30 

MAR 

81 

10 

APR 

81 

(8  SAC  TII ) 


*  PC  -  Participating  Command  (Two  test  image  interpreters) 


Detailed  Test  Schedule  (Continued) 


Activity 

#  Weeks 

Start  Date 

Part  2  Testing 

6 

13  APR  81 

Part  3  Test 

22 

1  JUN  81 

PCI  Training 

Test  Run  1 

4 

1  JUN  81 
15  JUN  81 

PC2  Training 

Test  Run  2 

4 

15  JUN  81 
29  JUN  81 

PC3  Training 

Test  Run  3 

4 

29  JUN  81 
13  JUL  81 

PC4  Training 

Test  Run  4 

4 

13  JUL  81 
27  JUL  81 

PC5  Training 

Test  Run  3 

4 

27  JUL  81 
10  AUG  81 

PCI  Training 

Test  Run  6 

4 

10  AUG  81 

2 A  AUG  81 

PC2  Training 

Test  Run  7 

4 

2 A  AUG  81 

8  SEP  81 

PC3  Training 

Test  Run  8 

4 

8  SEP  81 
21  SEP  81 

PC4  Training 

Test  Run  9 

4 

21  SEP  81 

5  OCT  81 

PC3  Training 

Test  Run  10 

4 

5  OCT  81 
19  OCT  81 

i 

i 

i 


End  Date 

22  MAY  81 

30  OCT  81 

12  JUN  81 
26  JUN  81 

26  JUN  81 
10  JUL  81 

10  JUL  81 
2 A  JUL  81 

2 A  JUL  81 
7  AUG  81 

7  AUG  81 
21  AUG  81 

21  AUG  81 
A  SEP  81 

A  SEP  81 
19  SEP  81 

18  SEP  81 
2  OCT  81 

2  OCT  81 
16  OCT  81 

16  OCT  81 
30  OCT  81 
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ANNEX  B 


OPERATIONAL  SUITABILITY 
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OPERATIONAL  SUITABILITY 

1.  MOE  or  Evaluation  Criteria 

The  test  objective  which  addresses  the  various  aspects  of  the  operational 
suitability  of  the  CPTB  will  be  evaluated  on  a  judgmental,  subjective 
basis.  No  specific  thresholds,  standards,  or  goals  will  be  established. 

2.  Methodology. 

To  form  a  sound  basis  for  judgment  of  the  systems  reliability,  maintain¬ 
ability,  and  supportability  records  will  be  maintained  throughout  the 
entire  test  period  including  pretest  preparation,  dry  run,  training  and 
test  conduct.  System  contractor  maintenance  personnel  will  complete 
Problem  Description  Reports,  Incident  Reports,  and  Trouble  Failure  Analy¬ 
sis  Reports  (see  examples  following)  covering  all  unscheduled  maintenance 
activities.  The  test  Director  and  Test  Manager  will  maintain  a  log  of 
system  usage/availability  documenting  all  test  utilization  time  on  the 
system  (including  preparation,  training,  etc.).  For  each  time  period  a 
record  will  be  kept  of  scheduled  time,  actual  sign  on/sign  off  time,  time 
lost  due  to  interruptions/failures,  etc.  The  log  will  also  contain  obser¬ 
vations/evaluations  of  system  operation  noting  any  anomalies,  operating 
difficulties,  etc. 

3.  Data  Management 

Copies  of  all  Maintenance  Contractor  reports  will  be  filed  in  the  test 
control  area.  The  Test  Manager  will  make  periodic  cross  checks  between 
the  Log  and  the  Failure  Report  file  to  assure  complete  data  capture. 

The  Failure  reports  will  be  reviewed  and  summarized  by  cognizant  develop¬ 
ing  command  personnel  and  the  results  prepared  and  submitted  to  the  Test 
Team  for  inclusion  in  the  test  reports. 

4.  Evaluation. 

The  Test  Team  Log  and  the  results  of  the  Failure  Report  summary  will  be 
reviewed  by  the  participating  command  representatives,  the  RADC  Test 
Director  and  the  Test  Support  Contractor  personnel  as  the  basis  for  the 
evaluative  conclusions/recommendations  related  to  various  potential  sys¬ 
tem  applications  (e.g.  further  test  bed  use  at  RADC  or  possible  future 
operational  command  utilization  of  the  system  or  particular  components). 

.1 

i 
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PROBLEM  DESCRIPTION  REPORT 

Initiator: _ Task _ 

Date  Submitted: _ 

Problem  with 

Subsystem/Function: _ _Module: 

Description  of  Problem: 


LU 

l/> 

PDR  No: 

Logged: 

z 

Priority: 

Effects/ Impacts: 


Proposed  Solution: 


CM  USE  ONLY 

Approved _  Disapproved 

Comments: 


Revision  to  Software: 
Revision  to  Document: 

Assigned  to: _ 

Close-out  Date: 


Deferred _ _ _ _ Analysis 


Status  Date: _ 

Initials: 


Eaton  Corporation 

Data  Systems  Services  Division 

6374  Arizona  Circle 

*  >  - 

INCIDENT  REPORT 

C  1445 

Los  Angeles,  CA  90045 

7  0 

Incident  Occu^ec' 

P  E  Not  ‘led 

$  OPARTS  DATA 


[5.1  Parti  Code  |5  2 


IIS3EHEE 


I 
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SOFTWARE  EVALUATION 


1.  General. 

Although  current  program  direction  does  not  call  for  the  CPTB  system  to 
become  operational,  much  can  be  learned  from  an  evaluation  of  the  software 
both  from  the  operability  and  maintainability  points  of  view.  The  proce¬ 
dures  employed  in  this  evaluation  will  be  drawn  from  the  ARTEC  Manual 
"Software  OT&E  Guidelines,  Volume  III,  Software  Maintainability  Evaluation 
Handbook",  1  August  1979,  supplemented  by  similar  procedures  directed  at 
operability  aspects. 

2.  MOE  or  Evaluation  Criteria. 

The  test  objective  concerned  with  software  maintainability  and  operability 
will  be  addressed  on  a  judgmental,  subjective  basis.  No  specific  standard 
comparison  (e.g.  a  predecessor  system)  exists.  No  specific  thresholds, 
standards,  or  goals  will  be  established.  The  evaluation  will  be  directed 
at  appraising,  on  a  judgmental  basis,  the  degree  to  which  the  system  soft¬ 
ware  could  be  supported  by  USAF  assets,  independent  of  the  developing 
contrac tor . 

3.  Methodology. 

The  methodology  to  be  employed  in  evaluating  the  CPTB  software  qualities 
of  maintainability  and  operability  is  based  upon  the  use  of  closed  form 
questionnaires  with  optional  written  comments.  The  questionnaires  are 
designed  to  determine  the  presence  or  absence  of  certain  desirable  main¬ 
tainability  attributes  in  each  major  software  component.  The  attributes 
employed  are:  (1)  Modularity,  (2)  Descriptiveness,  (3)  Consistency, 

(4)  Simplicity,  (5)  Expandability,  and  (6)  Instrumentation.  Both  the 
program  documentation  and  the  source  listing  of  each  major  software  com¬ 
ponent  will  be  reviewed  and  rated  for  each  of  the  attributes. 

A  parallel  investigation  of  the  operator  interface  for  each  relevant 
software  component  will  be  accomplished  from  an  operability  point  of  view. 
The  Operability  characteristics  to  be  rated  are:  (1)  Assurabi li ty , 

(2)  Controlabi lity ,  (3)  Work  Load  Resonability ,  (4)  Descriptiveness, 

(5)  Consistency,  and  (6)  Simplicity. 
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In  addition  to  the  detailed  examina t ion/rat ing  of  software  components  a 
complete  file  of  all  Baseline  Update  Requests  (see  example)  will  be  main¬ 
tained.  These  requests,  generated  by  system  contractor  personnel  during 
maintenance  activities,  will  be  duplicated  and  copies  delivered  to  the 
test  control  area  weekly.  The  copies  will  be  logged  in  and  tracked  to 
final  disposition. 

Software  maintainability  raters  will  be  drawn  from  SAC/AD  and  from  RADC. 
Software  operability  raters  will  include  the  four  SAC/AD  supplied  system 
operators  and  the  Til’s  from  Parts  2  and  3  of  the  Test. 

Duplicates  of  program  documentation  and  source  listings  will  be  supplied 
to  reach  evaluator,  by  RADC,  from  the  system  deliverable  package  follow¬ 
ing  Final  Acceptance  Testing.  Each  maintainability  evaluator  will  be 
assigned  a  system  familiarization  period  of  two  to  three  weeks  during 
which  there  will  be  an  opportunity  to  observe  (on  a  not-to-int er f ere 
basis)  system  operation  and  maintenance.  The  operability  evaluators  will 
have  received  formal  system  training  and  had  actual  experience  in  system 
operation. 

4.  Data  Management. 

Questionnaire  answer  sheets  completed  by  software  evaluators  will  be 
delivered  to  the  Test  Materials/Data  Control  (TM/DC)  Librarian  as  they 
are  completed.  The  TM/DC  Librarian  will  maintain  a  checklist  matrix. 
Evaluator  X  Software  component  to  insure  that  each  evaluator  rates 
each  component.  In  addition  a  composite  verbatim  transcript  of  evalua¬ 
tor  comments,  organized  by  individual  software  components,  will  be 
compiled.  When  all  evaluations  have  been  received  composite  ratings 
will  be  compiled.  Items  with  large  discrepancies  on  ratings  will  be 
noted  and  resolved  in  a  final  evaluator  conference.  Results  will  then 
be  compiled  and  incorporated  in  the  final  test  report. 
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BASELINE  UPDATE  REQUEST 


Initiator: _ 

Date  Submitted: 


BUR  No : _ 

Logged : _ 

Priority: 


I 


Update  to 

Subsystem/Function: _ Module: 


Note:  All  files  must  be  located  on  the  development  DB:.  Use  one  line  per  file. 
If  more  space  is  needed,  use  extra  sheet  of  blank  paper. 


SoftwareName. Ext  Module  ID  Entry  Point  TCR's 


QJ 

O  CO  ■*-> 

Task  ID  |  ^  ,  5 

0-0-010- 
o  <j  z:  rD 


Source 

UIC 


Documentation  to  be  updated: 
Reason  for  Change: 


Effect/Impact:(e.g. ,  anything  else  to  be  reMAC'd.reTKB'd?) 


PDR's  Closed: 

CM  USE  ONLY  ~ 

Approved _ _  Disapproved _  Deferred _ Analysis 

Comments: _ _ _ 


Scheduled  Update: 
Close-out  Date: 


Initials 
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ANNEX  E 


intelligence  and  threat 


(not  requited) 
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ANNEX  F 


DATA  MANAGEMENT 
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DATA  MANAGEMENT  PLAN 


1.0  General. 

The  procedures  for  the  management  of  the  anticipated  large  volume  of 
data  to  be  collected  in  the  course  of  the  planned  three  part  test  pro¬ 
gram  can  be  subdivided  in  two  ways;  first  in  terras  of  the  test  parts 
and  second  in  terms  of  specific  steps.  Each  test  part  requires  differ¬ 
ent  procedures.  In  all  parts  of  the  test  the  following  key  steps  must 
be  addressed,  following  initial  collection:  (1)  receipt,  (2)  valida¬ 
tion,  (3)  reduction,  (4)  retention,  (3)  processing,  and  (6)  analysis. 
The  primary  responsibility  for  these  activities  rests  with  the  344th 
SIW  Test  Director  although  the  actual  implementation  of  procedures  will 
be  the  duty  of  test  support  contractor  personnel;  the  on-site  test 
manager  and  the  Test  Materials/Data  Control  (TM/DC)  Librarian. 

2.0  Part  1.  Special  tests. 

There  are  two  major  data  sources  for  Part  1;  the  TA/DC  booklets  and  the 
TII  evaluation.  Handling  of  both  types  is  summarized  in  Figure  F-l. 

2.1  Receipt. 

Upon  completion  of  each  test  exercise  the  monitor  will  return  all  test 
materials  to  the  Test  Control  Area.  The  TM/DC  Librarian  will  check  in 
all  materials,  separating  and  refiling  the  monitor  materials,  (instruc¬ 
tions,  annotated  hard  copy,  etc.),  verifying  completion  of  the  TII 
evaluation  (if  required  by  the  schedule)  and  insuring  correct  labeling 
of  both  TA/DC  booklet  and  TII  evaluation  form.  Labeling  will  include: 
Test  Exercise  number,  date/ time,  TII  number,  and  monitor  name. 

2.2  Validation. 

The  TM/DC  Librarian  will  review  the  TA/DC  booklet  checking  for  the 
following:  (1)  each  data  entry  complete,  (2)  start  time  earlier  than 

stop  time,  (3)  notations  of  test  interupts,  system  failures,  etc.,  and 
(4)  general  legibility.  Anomalies  will  be  annotated  and  discussed  with 
the  monitor  at  the  earliest  opportunity.  (While  it  is  probably  not 
possible  in  most  instances  to  recover  missing  data,  feedback  to  moni¬ 
tors  should  minimize  future  repetitions  of  errors.)  Til  evaluation 


Figure  F-l  Data  Management  Schematic,  Part  1 
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forms  will  be  reviewed  for  completeness  and  legibility  only. 

2.3  Reduction. 

TA/DC  booklets  will  be  reduced  to  unclassified  answer  sheet  form  as 
soon  as  possible  following  receipt  and  validation.  Transcription  of 
times  and  answers  will  be  accomplished  by  the  TM/DC  Librarian.  Quality 
control  of  this  process  will  be  effected  by  periodic  spot  checks  by  the 
Test  Manager. 

Til  evaluation  forms  will  be  summarized  on  a  master  copy,  recording 
specific  item  by  item  responses  and  compiling  item  by  item  verbatim 
transcripts  of  comments. 

2.4  Retention. 

The  TA/DC  booklet  from  each  test  exercise  will  be  filed  in  chronolo¬ 
gical  order  within  each  test  exercise.  Test  exercise  files  will  he 
grouped  bv  specific  Part  1  subtest.  Each  answer  sheet  will  be  dupli¬ 
cated  and  one  copy  filed  with  the  TA/DC  booklet,  one  copy  forwarded 
to  IFR  State  College  for  processing  and  analysis.  Til  evaluation 
forms  will  be  separately  filed  in  chronological  order,  grouped  by 
Part  1  sub test. 

2.5  Processing. 

The  answer  sheets  will  be  processed  into  machine  readable  data  records 
and  100%  verified.  Each  data  record  will  retain  all  relevant  data 
labeling  including:  test  exercise  number,  TII  ID,  Monitor  ID,  frame 
number,  resolution  factor,  options  exercised,  etc.  TII  evaluation 
data  will  not  require  further  processing  beyond  the  completion  of 
the  summary. 

2.6  Analysis. 

Computer  analysis  of  the  TA/DC  data  utilizing  standard  statistical 
routines  will  be  accomplished  on  the  IFR  micro-processor  system  or  at 
the  Computation  Center  of  The  Pennsylvania  State  University,  depend¬ 
ing  on  data  volume  and  analysis  complexity.  Access  to  the  IFR  system 
is  assured  9ince  it  is  a  dedicated  facility.  Use  of  the  Penn  State 
facility  is  scheduled,  with  anticipated  run  turn  around  time  of  1-12 
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hours  depending  on  run  size.  Til  evaluation  form  data  will  be  anal¬ 
yzed  manually. 

3.0  Part  2,  DIAM  57-5  Reporting. 

There  are  four  major  types  of  data  collected  during  the  Part  2  tests: 
(1)  Time  data  from  the  Monitor  Forms;  (2)  Report  items/data  base 
entries  generated;  (3)  Target  materials  (H/C)  generated  on  CPTB,  and 
(4)  TII  evaluation  forms.  In  addition,  the  test  materials  used  in  the 
hard  copy  (HC)  condition  must  be  collected,  checked,  and  filed  for 
re-use.  Handling  of  the  four  data  types  and  the  test  material  is 
summarized  in  flow  chart  form  in  Figure  F-2. 

3.L  Receipt. 

At  the  end  of  the  first  test  exercise  period  of  each  test  day  the 
monitors  (1  soft  copy,  4  hard  copy)  will  return  all  materials  to  the 
Test  Control  Area.  The  TM/DC  Librarian  will  check  in  all  materials, 
separating  data  records  from  test  support  material  (i.e..  Monitor 
instructions,  reference  prints,  etc.)  and  HC  test  material.  Test 
support  material  and  HC  test  material  will  be  refiled  for  reuse. 
Completion  of  data  records  (i.e.  Monitor  time  sheets)  both  HC  and 
SC,  will  be  verified  and  checked  for  complete/correct  labeling. 

Target  materials  produced  on  CPTB  will  be  verified  for  completion/ 
labeling.  Printout  of  SC  generated  report  items/data  base  entries 
will  be  received  from  SC  monitor  and  verified  for  completion/ 
labeling.  Upon  completion  of  this  initial  data  receipt  the  TM/DC 
Librarian  will  retrieve  from  PACER  highspeed  printer  all  HC 
generated  report  items/data  base  entries,  verifying  completion/ 
labeling  of  outputs  from  all  four  HC  exercises.  At  the  end  of  test 
exercise  periods  2,  3,  and  4  of  each  test  day  the  SC  portion  of  the 
above  data  receipt  will  be  repeated.  At  the  end  of  the  entire 
Part  2  test  period  the  8  TTl’s  will,  as  a  group,  complete  the  evalu¬ 
ation  forms  and  return  them  to  the  Test  Control  Area  where  the 
TM/DC  Librarian  will  review  each  for  completeness  and  proper 
labeling. 
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Figure  F-2  Data  Management  Sehemalie,  Pari  1 
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3.2  Validation. 

Following  initial  check  in/receipt  of  data  records  from  each  test  exer 
cise  period  the  TM/DC  Librarian  will  review  all  Monitor  Time  Sheets, 
checking  for  the  following:  (1)  each  data  entry  complete;  (2)  start 
time  earlier  than  stop  time;  (3)  clear,  complete  notations  of  test 
interupts,  system  failures,  etc.,  and;  (4)  general  legibility. 
Anomalies  will  be  annotated  and  discussed  with  monitor  at  earliest 
opportunity  to  effect  correc tions/data  loss  recovery  (where  possible) 
and  to  prevent  recurrence  of  errors/omissions.  Report  iteras/data 
base  entries  and  target  materials  will  be  reviewed  in  an  attempt  to 
detect  systematic  errors/omissions  that  may  be  result  of  inadequate 
test  control/ instructions .  Any  such  anomalies  will  be  reported  to 
the  Contractor  On-site  Test  Manager  for  corrective  action.  Til  eval¬ 
uation  forms  will  be  reviewed  for  completeness  and  legibility  only. 

3.3  Reduction. 

It  is  anticipated  that  the  Monitor  Time  Sheets  will  be  in  the  form  of 
unclassified  keypunch  creation  sheets  requiring  no  t ranscription  prior 
to  processing.  Report  iteras/data  base  entries  and  Target  materials 
will  be  scored  against  school  solutions  and  completeness  and  accuracy 
data  recorded  on  unclassified  data  sheets  by  TM/DC  Librarian.  Quality 
control  of  the  scoring/recording  process  will  be  accomplished  by  repli 
cation  of  all  scoring/ recording  by  second  member  of  Test  Team  (pro¬ 
bably  HC  monitor)  and  comparison  verification.  Any  discrepanc ies  will 
be  resolved  by  joint  rescoring  of  subject  record. 

Til  evaluation  forms  will  be  summarized  by  the  TM/DC  Librarian  on 
master  copy,  recording  specific  item  by  item  responses  and  compiling 
item  by  item  verbatim  transcript  of  all  comments. 

Target  materials  will  be  accummulated  for  quality  evaluation  by  parti¬ 
cipating  command  representatives.  Each  participating  command  repres¬ 
entative  will  review  the  target  materials  (4  replications  of  each 
frame  used  judged  as  a  group)  using  a  standardized  rating  sheet.  The 
individual  ratings  will  be  collated  and  compiled  by  the  TM/DC  Librar¬ 
ian. 


70 


3.4  Retention. 

Monitor  Time  Sheets,  report  item/data  base  entry  printouts,  and 
Target  Materials  will  be  filed  by  test  exercise,  separated  by  Hard 
Copy/Soft  Copy  conditions.  Monitor  Time  Sheets,  report  item/data 
base  entry  score  sheets  and  Target  material  score  sheets  will  be  dup¬ 
licated.  The  originals  will  be  filed  in  the  Test  Control  Area;  copies 
forwarded  to  IFR,  State  College  for  processing  and  analysis.  Target 
material  quality  evaluations  will  also  be  duplicated  and  handled  sim¬ 
ilarly.  TII  evaluation  forms  will  be  filed  as  a  group  in  the  Part  2 
files . 

3.5  Processing. 

The  Monitor  Time  Sheets,  report  item/data  base  entry  score  sheets  and 
target  material  score  sheets  will  be  processed  into  machine  readable 
data  records  and  100%  verified.  Each  data  record  will  retain  all 
relevant  labeling  including:  test  exercise  number,  TII  ID,  Monitor  ID, 
frame  number,  installation  ID,  imagery  parameters,  etc.  Target  mater¬ 
ial  quality  ratings  and  TII  evaluation  data  will  not  require  processing 
beyond  original  collation,  compilation,  and  summarization. 

3.6  Analysis. 

Data  from  Monitor  Time  Sheets  and  report  item/data  base  entry  scoring 
will  be  analyzed,  using  standard  statistical  routines,  on  the  IFR 
micro-processor  system  or  at  the  Computation  Center  of  The  Pennsylvania 
State  University  via  an  established  RJE  link,  depending  upon  actual 
data  volume  and  analysis  complexity.  Access  to  the  IFR  system  is 
assured  since  it  is  a  dedicated  facility.  Use  of  the  Penn  State 
Center  facilities  is  scheduled  as  required  with  an  anticipated  job 
turn-around  time  of  i  to  12  hours  depending  on  run  size. 

Target  material  scoring  and  evaluation  data,  and  TII  evaluation  form 
data  will  be  analyzed  manually. 

4.0  Part  3,  Soft  Copy  Exploitation  Concepts. 

The  data  for  the  Part  3  test  arc  of  three  major  types;  (1)  interpre¬ 
tation  products;  (2)  observer  records,  and;  (3)  TII  evaluations.  The 
interpretation  products  are  report  items/data  base  entries  generated 
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under  all  three  test  conditions  which  will  be  evaluated  and  compared 
in  a  variety  of  ways  involving  both  subjective  ratings  and  objective 
scoring  in  relation  to  baseline  information  from  other  sources.  The 
observer  records  will  consist  of  task  timing  and  function  use  data  plus 
listings  of  f rames/instal lations  worked  in  the  CPTB  conditions.  The 
TII  evaluations  will  result  from  the  completion  of  questionnaires/ 
rating  sheets  by  each  participant  at  the  end  of  each  test  condition 
(CPTB  conditions  only) •  The  overall  data  management  flow  is  summarized 
in  Figure  F-3. 

.1  Receipt. 

The  CPTB  conditions  observer  will  twice  daily  (at  lunch  break  and  at 
and  of  shift)  deliver  to  the  TM/DC  Librarian  all  observation  forms, 
printouts  of  all  interpretation  products,  and  a  complete  inventory  of 
all  frames/installations  completed  during  the  test  segment.  The  HC 
baseline  observer(s)  will,  on  a  similar  schedule  deliver  all  observa¬ 
tion  forms  and  a  complete  listing  of  test  relevant  f rames/ ins ta 1 1  at i ons 
completed  during  that  test  segment.  Interpretation  products  resulting 
from  the  HC  baseline  condition  will  be  obtained,  by  the  TM/DC  Librarian 
from  the  PACER  high  speed  printer.  All  materials  will  be  cross  checked 
to  insure  complete  data  sets  (i.e.  matching  observer  form,  interpre¬ 
tation  product  and  frame/installation  listing).  Each  item  will  be 
checked  for  proper  labeling  at  time  of  receipt.  Labeling  will  include 
date,  time,  test  condition,  interpreter  ID,  Observer  ID,  and  frame 
number . 

2  Validation. 

The  major  validation  activity  related  to  these  data  will  be  a  review, 
by  the  TM/DC  Librarian  to  insure  complete  matching  data  sets  for  CPTB 
condition(s)  and  the*  corresponding  HC  baseline.  Each  observer  form 
will  also  be  reviewed  for  (1)  completeness,  (2)  start /stop  time  coher¬ 
ency,  notations  of  test  interrupts,  system  failures,  etc.  and  general 
legibility.  Anomalies  will  be  discussed  with  Test  Manager/ Ob server  at 
earliest  opportunity  to  prevent  recurrence  of  problems  and  resultant 
continued  data  loss.  TT I  evaluation  forms  will  be  reviewed  for  com- 
ple teness/legibili ty . 
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4.3  Reduction. 

No  immediate  scoring  or  transcription  will  be  required  for  observer 
forms  or  interpretation  products.  The  observer  form  (time  data  only) 
is  anticipated  to  be  unclassified.  These  forms  will  be  separated  from 
any  notes/additional  observations  which  may  require  security  classi¬ 
fication.  The  major  initial  activity  will  be  the  generation,  from  the 
CPTB  f rame/installat  ion  inventory,  of  an  HC  baseline  observer  checklist 
identifying  the  test  relevant  frames  to  be  observed  during  subsequent 
test  segments. 

TIT  evaluation  forms  will  be  summarized  on  a  master  copy,  recording 
specific  item  by  item  responses  and  compiling  item  by  item  verbatim 
transcripts  of  comments. 

4.4  Retention. 

Files  will  be  established  by  the  TM/DC  Librarian  such  that  each  data 
folder  will  contain  the  following  for  each  frame  completed:  (1)  Ori¬ 
ginal  CPTB  condition  observer  form,  (2)  Original  HC  baseline  observer 
form,  (3)  printout  of  CPTB  interpretation  products,  (4)  printout  of 
HC  baseline  interpretation  products,  (5)  Reference  dupe  pos  of  frame, 
and  (6)  CPTB  condition  observer  notes.  Data  folders  will  be  organi¬ 
zed  in  chronological  order  by  test  segment  in  terms  of  CPTB  interpre¬ 
ter  ID.  Copies  of  observer  time  sheets  will  be  forwarded  to  IFR, 

State  College  for  processing. 

4.5  Processing. 

Processing  of  the  Part  3  data  will  occur  in  several  stages.  The  obser¬ 
ver  time  data  will  he  directly  t  rails  fered  from  the  observer  sheets  to 
machine  readable  records  and  1002  verified.  The  interpretation  pro¬ 
ducts  will  require  sco~ ing/evaluat ion  by  participating  command 
representatives  at  the  Test  Facility.  Scores  and  evaluations  will  be 
recorded  on  unclassified  data  sheets.  These  sheets  will  be  copies, 
originals  filed  in  data  folders  and  copies  forwarded  to  IFR,  State 
College  to  be  transcribed  into  machine  readable  records  for  computer 
analysis.  Til  evaluations  will  not  require  processing  beyond  the 


original  summarization. 


Analys i s . 

Data  from  the  observer  forms  and  from  the  seoring/evaluat ion  of  inter¬ 
pretation  products  will  be  analyzed,  using  standard  statistical  routines 
on  the  IKR  micro-processor  system  or  at  the  Computation  Center  of 
The  Pennsylvania  State  University  via  an  established  RJF.  link,  depending 
upon  actual  data  volume  and  analysis  complexity.  Computer  access  is 
assured  as  indicated  in  sections  2.6  and  1.6. 

Til  evaluation  data  will  be  analyzed  manually. 


